Perspective

Dynamic Evaluation Approaches to Telehealth Technologies

and Artificial Intelligence (AI) Telemedicine Applications in
Healthcare and Biotechnology Organizations
Darrell Norman Burrell

Graduate and Doctoral Studies, Capitol Technology University, Laurel, MD 20708, USA;
[email protected]

Abstract: The COVID-19 pandemic has ushered in an unprecedented adoption and integration of
telehealth and artificial intelligence (AI) driven by telemedicine technologies into healthcare systems
worldwide. These innovations promise to revolutionize healthcare delivery by offering greater
accessibility, efficiency, and responsiveness to patient needs. However, the rapid deployment of these
technologies in response to the crisis has illuminated the imperative need for systematic evaluation
processes that comprehensively assess their operations and outcomes. This article underscores the
critical importance of developing rigorous evaluation frameworks tailored to the evolving landscape
of telehealth and AI-driven telemedicine technologies. The absence of standardized evaluation pro-
cesses presents multifaceted challenges including uncertainties regarding long-term efficacy, patient
safety, data security, and ethical considerations. Ensuring the responsible and effective integration of
telehealth and AI into healthcare systems requires adaptable, multidimensional evaluation mech-
anisms that align with clinical objectives and regulatory standards. Through an examination of
documents, procedures, policies, and best practices by regional hospitals, this article advocates for
developing evaluation processes that enable stakeholders to optimize the deployment of telehealth
and AI technologies fostering patient-centered care while addressing emerging challenges. In an
era marked by healthcare transformation, establishing robust evaluation frameworks emerges as a
paramount endeavor essential for realizing the full potential of telehealth and AI-driven telemedicine
in the post-COVID-19 healthcare ecosystem.
Citation: Burrell, D.N. Dynamic
Evaluation Approaches to Telehealth
Keywords: telehealth; telemedicine; mobile health; artificial intelligence; human-centered design;
Technologies and Artificial
biotechnology; health program evaluation; health administration; health technology
Intelligence (AI) Telemedicine
Applications in Healthcare and
Biotechnology Organizations. Merits
2023, 3, 700–721. https://doi.org/
10.3390/merits3040042 1. Introduction

Academic Editor: Wendy M. Purcell

The advent of the COVID-19 pandemic has propelled the healthcare landscape into
an era of rapid transformation marked by an unprecedented integration of telehealth and
Received: 1 October 2023 artificial intelligence (AI)-driven telemedicine technologies. This paradigm shift, driven by
Revised: 30 November 2023 the urgent need for innovative solutions, holds the promise of revolutionizing healthcare
Accepted: 4 December 2023 delivery, offering enhanced accessibility, efficiency, and responsiveness to patient needs.
Published: 6 December 2023
As these technologies become integral components of healthcare systems worldwide, the
need for systematic evaluation processes becomes increasingly apparent.
In the midst of this transformative wave, there exists a notable gap in research that
Copyright: © 2023 by the author.
squarely addresses the needs and perspectives of those entrenched in the day-to-day
Licensee MDPI, Basel, Switzerland. operations of healthcare and the broader world of practice. While considerable attention
This article is an open access article is rightfully devoted to the potential benefits of telehealth and AI-driven telemedicine
distributed under the terms and technologies, there is a pressing need to shift our focus to the systematic evaluation of their
conditions of the Creative Commons operations and outcomes, especially in real-world healthcare settings.
Attribution (CC BY) license (https:// This paper endeavors to underscore the critical importance of developing a rigorous
creativecommons.org/licenses/by/ evaluation framework tailored to the unique challenges and dynamics of the evolving land-
4.0/). scape of telehealth and AI-driven telemedicine technologies. The accelerated deployment

Merits 2023, 3, 700–721. https://doi.org/10.3390/merits3040042 https://www.mdpi.com/journal/merits

Merits 2023, 3 701

of these technologies in response to the crisis has brought to light the necessity for stan-
dardized evaluation processes. The absence of such frameworks introduces multifaceted
challenges including the uncertainties surrounding long-term efficacy, patient safety, data
security, and ethical considerations.
In this context, our exploration aims to advocate for the establishment of adaptable,
multidimensional evaluation mechanisms that align with clinical objectives and regulatory
standards. Beyond the theoretical promises of these technologies, the responsible and
effective integration of telehealth and AI into healthcare systems necessitates a nuanced
understanding of their impact on patient outcomes and healthcare operations.
Through a comprehensive examination of the current landscape, this article seeks
to contribute to the ongoing discourse by advocating for the development of evaluation
processes that empower stakeholders to optimize the deployment of telehealth and AI tech-
nologies. The ultimate goal is to foster patient-centered care while proactively addressing
emerging challenges.
In an era defined by healthcare transformation, the establishment of robust evaluation
frameworks emerges as a paramount endeavor. Only through tools, checklists, and real-
world approaches to the evaluation of operations and outcomes can we fully realize the
potential of telehealth and AI-driven telemedicine in shaping the post-COVID-19 healthcare
ecosystem based on real-world experiences and applications.

2. Problem Statement
The healthcare industry is undergoing a significant transformation propelled by the
advent of digital technologies and the incorporation of artificial intelligence (AI). According
to El-Yafouri [1], the global market for AI in healthcare exceeded $11 billion in 2021, and
projections anticipate a substantial increase to approximately $188 billion by 2030. This
exponential growth underscores the profound impact of AI on the healthcare landscape.
In the midst of this digital revolution that permeates every facet of our lives, healthcare
is experiencing a paradigm shift. New innovations are emerging to reshape the delivery of
healthcare services, elevating patient outcomes, and optimizing operational efficiency [1].
The continuous evolution of the healthcare sector is closely intertwined with the ongoing
transformation brought about by digital technologies.
In this dynamic landscape, digital advancements are not only altering the delivery
of care but also redefining the dynamics between patients and healthcare providers. As
patients increasingly turn to digital avenues for seeking care and physicians leverage
technology to deliver it, the traditional healthcare paradigm is being redefined [1]. This
metamorphosis is not merely a trend; it represents a fundamental restructuring of the
healthcare experience driven by the integration of digital technologies and AI.
The arrival of the COVID-19 pandemic has accelerated the adoption of telehealth and
AI-driven telemedicine technologies revolutionizing the landscape of healthcare delivery.
While these innovations offer immense potential to enhance accessibility, efficiency, and
quality of care, the rapid deployment of these technologies has underscored the pressing
need for robust evaluation processes to assess their operations and outcomes comprehen-
sively. This problem statement elucidates the critical importance of establishing systematic
evaluation frameworks tailored to the challenges and opportunities of integrating telehealth
and AI-driven solutions in the post-COVID-19 era.
The unprecedented expansion of telehealth services and AI applications in healthcare
has brought about an urgent imperative to ensure that these transformative technologies
align with clinical objectives, protect patient privacy, and yield positive patient outcomes.
Although the crisis necessitated rapid deployment, it has also introduced uncertainties
and knowledge gaps regarding these interventions’ long-term efficacy, safety, and ethical
implications. The absence of standardized evaluation processes tailored to the evolving
landscape of telehealth and AI-driven telemedicine technologies not only hinders the
realization of their full potential but also poses risks to patient care, data security, and
the sustainability of healthcare systems. As such, there is a compelling need for rigorous,
Merits 2023, 3 702

adaptable, and multidimensional evaluation frameworks that can guide stakeholders in

optimizing the integration of telehealth and AI in healthcare while ensuring that patient-
centered goals and regulatory standards are met. This article explores and advocates for
developing such evaluation processes imperative for advancing the responsible, effective,
and equitable implementation of telehealth and AI-driven telemedicine technologies in the
post-COVID-19 healthcare landscape.

3. Purpose
The paper intends to explore evaluation approaches to telehealth technologies and
emerging applications of AI in telehealth. This research aims to influence the world of practice
and healthcare leaders as they make strategic and contingent plans to implement AI.

4. Design/Methodology/Approach
This inquiry is intended to influence the practice which matters significantly to those
in leadership decision-making roles looking for actionable guidance as new healthcare
technologies emerge. Collecting data for this conceptual paper was aimed at developing
critical evaluation approaches and guidelines by benchmarking procedures and evalua-
tion telehealth technology documents and emerging AI telehealth documents of the two
largest VA hospitals in the state of Virginia and the state of West Virginia (four total). The
process involved a comprehensive review of the hospitals’ official documents, policies, and
guidelines. The scrutinized documents exclusively encompassed the process and program
evaluation materials pertaining to hospital protocols, quality assurance assessments, pa-
tient feedback records, and regulatory compliance documentation. All are germane to the
realm of telehealth technologies and the applications of artificial intelligence therein. This
method of document analysis was instrumental in engendering an exhaustive comprehen-
sion of the extant procedural frameworks and guidelines with the overarching objective
of amalgamating divergent methodologies into a unified, comprehensive approach for
critical evaluation that espouses pertinent and evidence-based criteria as corroborated by
esteemed scholars such as Karppinen and Moe [2], and Bowen [3]. Document analysis
can successfully and effectively function in a central role as the solitary research method-
ology in certain instances [4–6]. In such cases, the use of pre-existing documents affords
researchers unparalleled access to a wellspring of data serving as the preeminent resource
for the successful fruition of research projects as validated by the scholarly contributions of
Karppinen and Moe [2], and Bowen [3].
Applied research approaches play a pivotal role in addressing and solving complex
real-world problems and offering a direct bridge between theory and practice [4–6]. The
importance of applied research approaches lies in their capacity to generate actionable
insights and practical solutions that have a tangible impact on individuals, organizations,
and societies at large [4–6]. Unlike purely theoretical research, applied research is grounded
in the practical context allowing for the testing of hypotheses and the development of
evidence-based strategies [6]. This form of research provides invaluable guidance for
decision-makers across various domains ranging from healthcare and business to public
policy and technology, enabling them to make informed choices, optimize resources, and
navigate the multifaceted challenges of our rapidly changing world. Ultimately, the sig-
nificance of applied research lies in its ability to transform knowledge into action to foster
innovation, progress, and meaningful change in the pursuit of solutions to some of the
most pressing and intricate problems of our time [4–6].
Practical and applied approaches to topics are pivotal in building the foundation
for future empirical and quantitative studies [4–6]. These approaches provide a tangible
connection between theory and real-world scenarios, offering researchers invaluable in-
sights into the practical implications of their hypotheses [4–6]. By delving into practical
applications, researchers gain a deeper understanding of the challenges, nuances, and
complexities that may not be apparent in the theoretical frameworks alone [6]. This experi-
ential knowledge serves as a vital precursor to the design and execution of empirical and
Merits 2023, 3 703

quantitative studies guiding researchers to formulate relevant research questions, develop

robust methodologies, and interpret data contextually and meaningfully [6,7].
Furthermore, practical and applied approaches often serve as a litmus test for the
feasibility and relevance of research inquiries, helping researchers refine their focus and
prioritize areas of investigation with the most significant potential for impact [6,7]. In
essence, the synergy between practical exploration and empirical analysis forms a dynamic
continuum in research wherein the former informs and enriches the latter, ultimately
advancing the depth and applicability of scientific inquiry [4–6].
Document analysis is an empirical research approach that is often underutilized and
underestimated, and it offers a systematic and structured method for extracting valuable
insights from existing documents contributing to the empirical foundation of applied
research [8]. Document analysis involves the comprehensive review and systematic ex-
amination of documents such as reports, policy documents, archival records, and other
textual data to extract meaningful information, patterns, and insights [8]. It operates on the
premise that documents, as repositories of human knowledge and actions, contain valuable
empirical data that can be rigorously analyzed to inform research inquiries [8]. Document
analysis is rooted in empirical inquiry as it relies on the tangible artifacts of human activity
to draw conclusions and generate knowledge [8].

5. Key Principles of Document Analysis

Document analysis follows a structured and systematic approach which includes
defining research questions, selecting relevant documents, and developing coding schemes.
To enhance empirical rigor, document analysis often employs triangulation by cross-
referencing information from multiple documents or sources [8]. An exploration of doc-
uments emphasizes understanding the context in which the documents were created
allowing researchers to interpret findings within their historical, social, and organizational
contexts [8]. Document analysis strives for transparency and reproducibility, ensuring that
the research process can be replicated and verified by other scholars [8].

6. Originality/Value
While theoretical discussions are valuable, the true fulfillment of these technologies’
potential necessitates a shift toward practical tools, checklists, and real-world approaches.
For healthcare leaders navigating unfamiliar territory, these tangible resources will serve
as guiding lights providing structured pathways for the evaluation of operations and
outcomes. It is in these practical applications that the transformative power of telehealth
and AI-driven telemedicine will manifest concretely.
For those with limited exposure to these technologies, the starting point lies in the
systematic exploration of existing evaluation frameworks. This involves understanding the
intricacies of assessing operations and outcomes, identifying key performance indicators,
and establishing benchmarks for success. Furthermore, embracing a learn-as-you-go
approach can demystify the complexities, allowing healthcare leaders to incrementally
incorporate telehealth and AI into their operational landscape.
Equally crucial is the creation of user-friendly guides and toolkits tailored for health-
care leaders with varying levels of expertise. These resources should provide step-by-step
instructions, best practices, and case studies offering practical insights that bridge the
gap between theory and application. Collaborative platforms for knowledge sharing and
peer-to-peer learning can further enhance the accessibility of these resources, fostering a
supportive community for healthcare leaders navigating the integration of telehealth and
AI technologies.
In essence, our journey toward unlocking the full potential of telehealth and AI-driven
telemedicine involves not just theoretical discourse but also a commitment to practical, action-
able steps. By providing accessible tools and frameworks, we empower healthcare leaders
with limited exposure to navigate these transformative technologies confidently, ensuring that
the promises of enhanced patient care and operational efficiency become a tangible reality.
Merits 2023, 3 704

This research is valuable because it is intended for real-world healthcare managers

who often face challenges with solving actual organizational problems through the abstract
nature of theoretical and highly quantitative research studies. This paper represents a
pioneering practice endeavor characterized by its profound originality and novelty by
developing evaluation approaches and recommend steps. The originality of this study
lies in its exploration of the swift and unprecedented adoption of telehealth technologies
in response to the urgent demands imposed by the COVID-19 pandemic. With the rapid
proliferation of telehealth, healthcare systems across the globe have been compelled to
embrace these technologies as a primary means of delivering care, often bypassing the
conventional thoroughness of evaluation methods and processes. This research endeavors
to shed light on this remarkable juncture where necessity expedited innovation and health-
care practices underwent a paradigm shift characterized by adaptability and innovation
amidst uncertainty.
Furthermore, this research aims to distill invaluable insights from this unique context,
thereby informing future strategies for the integration of telehealth into routine healthcare
delivery. By traversing this uncharted territory, it seeks to foster a deeper understanding
of the intricate interplay among technology, healthcare, and crisis response, ultimately
contributing to the reservoir of knowledge essential for building resilient healthcare systems
and safeguarding the well-being of patients in a world that continues to grapple with
unforeseen challenges.

7. Findings
The rapid and unprecedented expansion of telehealth services and the integration
of AI applications in healthcare have underscored the urgent necessity to align these
transformative technologies with clinical objectives, safeguard patient privacy, and ensure
positive patient outcomes. While the crisis-driven acceleration of these innovations was a
response to immediate needs, it has simultaneously introduced uncertainties and created
knowledge gaps about their long-term effectiveness, safety, and ethical implications. The
absence of standardized evaluation mechanisms tailored to the ever-evolving terrain of
telehealth and AI-driven telemedicine technologies not only impedes the realization of their
full potential but also poses risks to patient care, data security, and the sustainability of
healthcare systems. Consequently, a compelling imperative exists for developing rigorous,
adaptable, and multifaceted evaluation frameworks. Such frameworks are indispensable
tools for guiding stakeholders in optimizing the seamless integration of telehealth and AI
technologies into healthcare systems, all while ensuring that patient-centered objectives
and regulatory standards remain at the forefront of this transformative process. This article
explores and advocates for establishing these crucial evaluation processes and imperative
prerequisites for the responsible, effective, and equitable implementation of telehealth and
AI-driven telemedicine technologies in the post-COVID-19 healthcare landscape.

8. Research Limitations
The research project’s primary method, document analysis, offers valuable insights
into the selected research topic, specifically the evaluation of telehealth technologies and
artificial intelligence applications within healthcare settings. However, it is imperative to
acknowledge certain inherent limitations that may impact the scope and generalizability of
the findings.
This inquiry exclusively relies on pre-existing documents for data collection from only
four government hospitals in a limited geographical area in the United States, which may
introduce limitations related to data completeness and relevance. The documents available
for analysis may not encompass all pertinent aspects of the research topic and thus omit
information that could provide a more comprehensive understanding of the subject matter.
Additionally, the research relies on the accuracy and reliability of the documents themselves,
and any errors or biases present in the original documents could be perpetuated in the analysis.
Merits 2023, 3 705

9. Overview
Telehealth uses telecommunications technology to provide remote healthcare services,
consultations, and patient support. It encompasses various healthcare activities including
consultations, monitoring, education, and care administration conducted remotely via
video conferencing, phone calls, mobile apps, and other digital platforms [9]. Telehealth
has gained significant traction recently, driven by technological advancements and the need
for accessible and convenient healthcare, especially in remote or underserved areas [9]. AI
technologies are increasingly integrated into telehealth systems, revolutionizing patient
engagement, treatment, and diagnosis processes [10–13].
Here is an overview of how AI-powered telehealth can be used in each of these areas:

9.1. Engaging Patients

Personalized Health Education: AI can tailor health education materials and messages
to individual patients based on their medical history, preferences, and health goals [10–13].
This promotes patient engagement and adherence to treatment plans. Chatbots and Virtual
Assistants: AI-driven chatbots and virtual assistants can engage patients by providing real-
time responses to inquiries, appointment scheduling, medication reminders, and general
health guidance [10–13].

9.2. Treating Patients

Remote Monitoring: AI-enabled wearable devices and sensors can continuously mon-
itor patients’ vital signs and transmit data to healthcare providers. AI algorithms can
analyze these data to detect anomalies or trends, enabling early intervention [10–13].
Decision Support: AI-powered clinical decision support systems assist healthcare
providers in making accurate diagnoses and treatment recommendations by analyzing
patient data and the medical literature [10–13].
Medication Management: AI telehealth tools can help patients manage medications
by providing dosage reminders, drug interaction alerts, and adherence tracking [10–13].

9.3. Diagnosing Conditions

Medical Imaging Analysis: AI algorithms can analyze medical images (e.g., X-rays,
CT scans, and MRIs) with high accuracy to assist radiologists in detecting and diagnosing
conditions such as tumors, fractures, and abnormalities [10–13].
Natural Language Processing (NLP): NLP-powered AI can transcribe and analyze spo-
ken or written patient information during telehealth consultations. It assists in extracting
relevant medical history, symptoms, and insights for diagnosis [10–13].
Symptom Checker and Triage: AI-driven symptom checkers can evaluate patient-
reported symptoms, provide preliminary diagnoses, and recommend appropriate next
steps such as scheduling a telehealth appointment or seeking in-person care [10–13].
AI-enhanced telehealth technologies are valuable in improving healthcare accessibil-
ity, streamlining diagnosis and enhancing patient engagement. They enable healthcare
providers to reach a broader patient base, offer more personalized care, and make more
accurate diagnoses, ultimately leading to better health outcomes and increased patient
satisfaction. Additionally, AI continuously learns and improves, making it a valuable tool
in the rapidly evolving field of telehealth [10–13].
Telehealth, mobile health, and remote patient monitoring technologies have been
growing in popularity as they provide a convenient and cost-effective way of delivering
patient care [14]. Telehealth, which uses AI or Internet of Things (IoT) technologies, has the
potential to revolutionize healthcare [12,15,16]. However, ethical considerations must be
taken into account to ensure that the unique needs of patients, particularly those whose first
language is not English or those who are visually or hearing impaired, are met [12,15,16].
As these technologies become more widely adopted, it is important to remember that
English is not the first language for many patients and other patients have visual and hearing
disabilities related to telehealth, mobile health, and remote patient monitoring with the help
Merits 2023, 3 706

of AI or the IoT. For telehealth to be used ethically, healthcare providers must consider the
challenges associated with providing care to patients who do not speak English or have
vision or hearing impairments [12,15,16]. For instance, AI and IoT technologies may require
new interfaces such as voice recognition that are not accessible to patients who are deaf or
have limited vision. Furthermore, language barriers may impede communication between
health providers and patients, leading to delays in diagnosis and treatment [12,15,16].
To address these challenges, healthcare providers must consider the unique needs of
patients and design systems that are accessible to all regardless of language or disability.
For example, healthcare providers can use AI and IoT technologies to develop user-friendly
interfaces accessible to all patients including those with language barriers or who are
visually or hearing impaired. Additionally, healthcare providers should consider using
multilingual interfaces or voice-recognition software to ensure that language barriers do not
hinder communication between the provider and patient. By taking these steps, healthcare
providers can ensure that telehealth is used ethically and that patients receive the care they
need regardless of their language or disability [12,15,16].
In addition to considering the unique needs of patients, healthcare providers must also
consider the potential risks associated with using AI and IoT technologies in telehealth [11].
For instance, data security is a significant concern as AI and IoT technologies may lead to
collecting and storing large amounts of sensitive patient data. Using these technologies
may also lead to privacy concerns as information about a patient’s health may be shared
with third parties without their consent. To address these issues, healthcare providers must
ensure appropriate safeguards to protect patient data and privacy [9].
The use of key theories is a useful way to understand the adoption and successful use
of telemedicine and AI telehealth technologies within healthcare organizations [9]. This
study employs this method by outlining critical elements of each theory and elucidates how
these theoretical frameworks can shed light on the decision-making processes of patients
and healthcare providers in adopting or resisting these innovative technologies.

9.4. Managing Diabetes and Heart Conditions

AI has made significant contributions to mobile health (mHealth), remote monitoring, and
telemedicine, particularly in the context of managing diabetes and heart conditions [17–19].
Here are some notable uses.
Early Detection and Diagnosis
AI algorithms can analyze data from wearable devices, such as smartwatches and
continuous glucose monitors, to detect early signs of heart conditions and fluctuations
in blood glucose levels indicative of diabetes. This enables timely intervention and
preventive measures.
Personalized Treatment Plans
AI can analyze vast amounts of patient data considering individual health records,
lifestyle factors, and real-time monitoring data. This allows for the creation of personalized
treatment plans for patients with diabetes and heart conditions optimizing medication
regimens and lifestyle recommendations.
Remote Monitoring
Wearable devices equipped with AI capabilities enable continuous remote monitoring
of vital signs, glucose levels, and other relevant health parameters. Healthcare providers
can receive real-time data allowing for prompt adjustments to treatment plans and early
identification of potential issues.
Predictive Analytics
AI algorithms can predict potential complications related to diabetes and heart conditions
by analyzing historical patient data. This helps healthcare providers anticipate and address
risks before they escalate, improving patient outcomes and reducing emergency situations.
Merits 2023, 3 707

Medication Adherence
AI-powered mobile applications can send reminders for medication adherence and
provide educational resources to enhance the patient’s understanding of treatment plans.
These apps can also track medication usage patterns and notify healthcare providers if
there are deviations from the prescribed regimen.
Teleconsultations
AI-driven chatbots or virtual health assistants can facilitate teleconsultations to provide
patients with instant access to healthcare information and guidance. This is particularly
useful for routine check-ups, medication adjustments, and general inquiries, enhancing
patient engagement and reducing the burden on healthcare systems.
Continuous Glucose Monitoring (CGM)
AI algorithms integrated with CGM devices can analyze glucose trends and patterns,
providing valuable insights into a patient’s diabetes management. This information can
be shared with healthcare providers to enable informed decision-making and proactive
adjustments to insulin therapy.
Rehabilitation Support
For patients recovering from heart conditions, AI-based applications can deliver
personalized rehabilitation plans including exercise routines and dietary recommendations.
These plans can be adapted based on the individual’s progress and real-time health data.
Data Security and Privacy
AI plays a crucial role in ensuring the security and privacy of patient data in mobile
health applications. Advanced encryption and authentication mechanisms backed by AI
algorithms help safeguard sensitive health information.
The integration of AI into mobile health, remote monitoring, and telemedicine for
diabetes and heart conditions enhances early detection, personalization of care, remote
monitoring, and overall patient outcomes [17–19]. It also contributes to more efficient and
effective healthcare delivery.

10. Tele-ICU and Tele-Stroke Programs

10.1. Tele-ICU
Tele-ICU and Tele-Stroke programs exemplify the transformative impact of AI in
enhancing healthcare delivery, particularly in critical care scenarios. The Tele-ICU initiative
employs AI-powered technologies to connect specialty providers with intensive care unit
(ICU) teams through video conferencing [20,21]. Equipped with TV screens, cameras, and
call-buttons, Tele-ICU rooms facilitate seamless communication between on-site healthcare
providers and remote specialists. This technology enables virtual consultations where
providers can engage with patients, review electronic health records, and document ob-
servations about health conditions. AI algorithms may assist in the real-time analysis of
patient data offering valuable insights to the on-site care teams. The collaborative efforts be-
tween local providers and remote specialists supported by AI contribute to more informed
decision-making during critical points of patient care. This integrated approach ensures
that treatment plans align with the specific needs of each patient [20,21].

10.2. Tele-Stroke Program

In a similar vein, the Tele-Stroke program uses AI-driven solutions to address the
challenges faced by healthcare facilities, especially in rural areas lacking continuous acute
stroke coverage [22,23]. Delivering timely specialized stroke care is crucial for improving
patient outcomes. Through video tools, Tele-Stroke neurologists remotely examine patients
and leverage AI for a comprehensive analysis of neurological indicators. This virtual
consultation allows timely collaboration with local healthcare providers, aiding in the swift
diagnosis and recommendation of treatment plans [22,23]. The use of AI in this context not
Merits 2023, 3 708

only expedites the decision-making process but also enhances the accuracy of diagnostic
evaluations, ultimately increasing the likelihood of successful patient recovery.
Both Tele-ICU and Tele-Stroke applications underscore the potential of AI to bridge ge-
ographical gaps and bring specialized medical expertise to areas where they may originally
be limited. These programs showcase how AI technologies empower healthcare providers
to deliver more efficient and effective care, particularly during critical junctures in patient
treatment. The integration of AI into telemedicine initiatives exemplifies a paradigm shift in
healthcare, leveraging advanced technologies to optimize decision-making, improve patient
outcomes, and extend the reach of specialized care to diverse and underserved populations.

11. English as a Second Language

Telehealth, mobile health, and remote patient monitoring technologies can quickly
increase access to health care for those who cannot access a medical facility [9]. However,
these technologies can be a double-edged sword for patients who cannot read or understand
English. Technology can be a barrier to accessing care for these patients as the instructions
and user interfaces are often written in English [12,15,16].
Language barriers can be especially problematic for patients who need urgent care
and do not have access to an interpreter [9]. To address this issue, providers should ensure
that all patient-facing materials, such as instructions and user interfaces, are available in the
patient’s native language [12,15,16]. The patient should also be provided with an interpreter
if they cannot understand the instructions or the user interface. Additionally, providers
should ensure that all patient data are stored securely and privately even when transmitted
in a foreign language [12,15,16]. AI telehealth technologies can be instrumental in address-
ing the unique healthcare needs of patients for whom English is a foreign language (EFL).
These technologies can facilitate effective communication, provide language assistance, and
enhance the overall healthcare experience [13,15]. Here are some AI telehealth technologies
that can be used to help EFL patients:

11.1. Multilingual Chatbots and Virtual Assistants

AI-powered chatbots and virtual assistants can be used to provide instant multilingual
support to EFL patients. They can answer basic medical questions, schedule appointments,
and offer guidance on common health concerns in various languages.

11.2. Real-Time Language Translation Services

AI-driven language translation tools integrated into telehealth platforms can be used
to enable real-time translation of spoken or written communication between patients and
healthcare providers. This ensures precise and accurate communication regardless of any
language barriers.

11.3. Cultural Sensitivity Training Modules

AI can help create culturally tailored training modules for healthcare providers to
enhance their understanding of diverse patient populations. This helps providers commu-
nicate more effectively and respectfully with EFL patients.

11.4. Automatic Medical Record Translation

AI can automatically translate medical records and documentation from English to
the patient’s preferred language. Translational tools ensure that patients can access their
healthcare information in a language they understand.

11.5. Voice Recognition and Speech-to-Text

AI-driven speech recognition and transcription tools can be used to convert spoken words
into text in real time. Speech recognition tools aid in capturing accurate patient information
during telehealth consultations, which can then be translated or reviewed as needed.
Merits 2023, 3 709

11.6. Telehealth Platforms with Multilingual Interfaces

AI-powered telehealth platforms can offer multilingual user interfaces, making it
easier for EFL patients to navigate and use the technology. AI-powered telehealth platforms
include appointment scheduling, prescription refills, and access to educational resources.

11.7. AI-Powered Language Assessment and Learning Tools

AI can assess the language proficiency of EFL patients and provide tailored language
learning resources. These tools can help patients improve their English language skills and
better understand medical information.

11.8. Multilingual Medication Reminders and Health Alerts

AI-driven medication reminder apps and health alert systems can be used to deliver
notifications and instructions in multiple languages, ensuring that EFL patients receive
critical health information accurately.

11.9. Cultural Competency Algorithms

AI algorithms can analyze patient data and provide healthcare providers with insights
into cultural factors that may impact a patient’s health behavior and decision-making. Tools
that consider cultural factors help by providing more culturally sensitive care.

11.10. AI-Enhanced Tele-Interpreting Services

Tele-interpreting services with AI enhancements can connect patients with trained
interpreters in real time during telehealth consultations. AI can assist interpreters with
medical terminology and context to ensure accurate communication.
These AI telehealth technologies bridge language barriers and promote cultural sen-
sitivity, making healthcare more accessible and inclusive for EFL patients. By improving
communication and understanding, these technologies enhance the quality of care and
patient outcomes [13,15].

12. Visual and Hearing Disabilities

Patients with visual or hearing disabilities face challenges using telehealth, mobile
health, and remote patient monitoring technologies [12,15,16]. For example, if the patient
is visually impaired, they may be unable to use the technology if it is not designed with
accessibility in mind [9]. Additionally, if the patient is hearing-impaired, they may need help
understanding the instructions or user interface if it is provided in audio form. To address
this issue, providers should ensure that all patient-facing materials, such as instructions
and user interfaces, are designed with accessibility [9]. Assisting technology tools include
providing alternative forms of media such as text, video, and audio to accommodate
patients with visual or hearing disabilities [9].
Additionally, providers should ensure that the patient is equipped with the necessary
technology such as an audio amplifier or a Braille display to ensure that they can fully access
the technology [9]. Telehealth, mobile health, and remote patient monitoring technologies
can increase access to care for vulnerable populations [9]. However, these technologies
can also be a barrier to accessing care if not designed with accessibility [9]. Healthcare
providers must be mindful of the potential for AI and IoT technologies to introduce bias
into the healthcare system [12,15,16]. For example, AI-driven decision-making may exclude
specific patient populations such as those from lower socio-economic backgrounds or
minority groups [9]. To avoid this, healthcare providers must ensure that the data used
to train AI and IoT systems are diverse and that algorithms are tested for bias before
deployment [12,15,16].
Using AI and IoT technologies in telehealth can revolutionize healthcare and improve
patient outcomes [12,15,16]. However, ethical considerations must be considered to ensure
that the unique needs of patients, particularly those whose first language is not English or
those who are visually or hearing impaired, are met [9]. By taking these steps, healthcare
Merits 2023, 3 710

providers can ensure that telehealth is used ethically and that patients receive the care they
need regardless of their language or disability.
AI telehealth technologies can be crucial in providing accessible healthcare services to
patients with visual and hearing disabilities. AI telehealth technologies aim to enhance com-
munication, support remote medical consultations, and ensure that healthcare information
is accessible [13,15].
AI telehealth technologies for patients with visual and hearing disabilities should
include the following:
Voice-Activated Assistants and Screen Readers
AI-powered voice-activated assistants, like Amazon’s Alexa or Google Assistant, can
help patients with visual impairments by providing information and assistance through voice
commands. Screen readers with AI capabilities can convert text on screens into synthesized
speech, allowing visually impaired patients to access digital healthcare content [24].
AI-Enhanced Image Recognition
AI algorithms can describe images and visual content to patients with disabilities. For
example, a smartphone app can use AI to describe medical images such as X-rays or MRIs
to patients.
Braille Displays with AI Integration
Braille displays enhanced with AI can provide real-time translations of digital text into
Braille so visually impaired patients can read healthcare information and communicate
with healthcare providers.
Accessible Telehealth Platforms
Telehealth platforms with AI-driven accessibility features can ensure that patients
with visual disabilities can navigate and use the platform easily. These platforms include
voice-guided interfaces and screen reader compatibility.
Real-time Captioning and Transcription Services
AI-powered real-time captioning services can provide automated captions during
telehealth video calls, making spoken information accessible to patients with hearing
impairments. Speech-to-text transcription services can convert spoken words into written
text for easy reading.
Sign Language Recognition and Translation
AI can recognize sign language gestures and translate them into text or spoken lan-
guage for healthcare providers. These technologies can enable real-time communication
with sign language users.
Speech Enhancement and Noise Reduction
AI algorithms can improve audio quality during telehealth consultations by reducing
background noise and enhancing speech. These tools are particularly beneficial for patients
with hearing aids or cochlear implants.
Text-Based Communication Tools
AI-driven chatbots and text-based communication tools can be used to facilitate written
communication between patients and healthcare providers, ensuring that information is
effectively conveyed.
Accessible Telehealth Interfaces
Telehealth platforms with AI-driven accessibility features can be used to offer visual
cues and notifications to alert patients with hearing disabilities about upcoming appoint-
ments or essential information.
Remote Sign Language Interpreting Services
Merits 2023, 3 711

AI-powered tele-interpreting services can be used to connect patients with sign lan-
guage interpreters during telehealth consultations, making communication seamless and
accessible. By leveraging these AI telehealth technologies, healthcare providers can ensure
that patients with visual and hearing disabilities have equal access to healthcare services,
effective communication with healthcare professionals, and the ability to make informed
decisions about their healthcare [13,15]. These technologies are pivotal in promoting
inclusivity and improving the overall quality of care for individuals with disabilities.
Evaluating the Implementation of New Telehealth Technologies
Noel and Fabus [25] outline some areas of evaluation for the implementation of
telehealth, including:
The participation of patients: Patient participation may be impacted as telehealth
continues to spread along the spectrum of healthcare. Patients may keep track of their
medical issues, results, and general well-being using several technologies, and they can also
stay in communication with their physicians to participate more actively in their medical
care. These tools allow patients to engage more thoroughly with their medical state [25].
With the numerous advantages of using evidence-based best practices available via
telehealth when using a particular kind of telehealth technology, it is the patient’s re-
sponsibility to self-monitor and ensure that they maintain open communications with
their physicians. This active collaboration has the potential to enhance the active man-
agement of symptoms, which may lead to fewer trips to the emergency department and
hospitalizations in general. This would be a positive outcome. The implementation of
telehealth demonstrates explicitly the potential to reduce costs, rates of hospitalization, and
readmissions related to chronic illnesses [25].
The coordination of care: The coordination of care for patients who have complicated
care requirements (e.g., patients who have numerous chronic diseases, patients who need
rehabilitative services, or patients who need specialty care) is an essential component of
treatment. Patients who have complex care needs include those making the transition of
care from an outpatient setting to an inpatient setting and from an inpatient setting to a
long-term care nursing facility or other clinical settings; telehealth may be able to facilitate
communication, the sharing of information, and the making of joint decisions. An objective
evaluation of telehealth’s capacity to support such coordination would be a prerequisite for
determining whether or not a telehealth program is successful and how it affects patients’
health outcomes [25].
Traveling to the appointment impacts: Patients should not think of travel as just an
accrued benefit for cost savings and convenience; instead, patients should use it to deter-
mine whether or not the use of telehealth has led to the correct diagnosis and appropriate
follow-up care to reduce the need for further travel. Evaluations focus on using travel
aspects more comprehensively to determine whether telehealth was responsible. The time
the patient saves during the first appointment is considered, but it should also consider the
outcomes. If the patient receives a negative diagnosis, there will be no need for a second
visit in person [25].
The promptness of the care: There is a correlation between receiving care promptly
and having better health results. A late diagnosis and treatment, missing abnormalities
that showed on screening, and patients with correctly identified abnormalities who did not
have a follow-up with a physician were some factors that led to worse survival rates with
conditions such as cancer. In addition, a delayed diagnosis following an initial screening
can lead to worse survival rates among individuals with certain forms of cancer (e.g., lung
cancer) and difficulties due to chronic illness [25].
Information that can be acted upon: For care team members to use the information
provided by telehealth technology during the initial visit, this information must be action-
able. This information may contain data that enable a physician to diagnose and treat the
patient and offer any necessary follow-up treatment. Additionally, this information may
permit the provider to give any necessary follow-up care [25].
Merits 2023, 3 712

Evaluation Is Important
The integration of technology into healthcare has witnessed an exponential rise from
electronic health records and telemedicine to artificial intelligence and wearable devices.
While these innovations hold the promise of improved patient outcomes, increased effi-
ciency, and enhanced healthcare delivery, they also bring forth a need for robust program
evaluation methodologies to guide their implementation [26]. This imperative stems from
the recognition that the mere adoption of technology does not guarantee its effectiveness or
alignment with healthcare objectives [27]. Rather, it is the systematic assessment guided by
inquiry into processes, policies, programs, procedures, and user experiences that elucidates
the true impact of these technologies on healthcare systems and patient care [26].
The confluence of technology and program evaluation presents a remarkable oppor-
tunity to propel healthcare into a new era of excellence and efficiency [26]. As healthcare
organizations navigate the complexities of digitalization, program evaluation emerges as
an indispensable compass guiding the effective use of technology to achieve healthcare
goals [27]. By systematically inquiring into processes, policies, programs, procedures,
and user experiences, healthcare evaluators can illuminate the path toward a technology-
enhanced healthcare ecosystem that optimally serves the needs of providers and patients
alike [27]. The significance of technology-driven program evaluation in healthcare is not
merely a response to the digital age; it is a commitment to harnessing the full potential of
technology to elevate the quality, accessibility, and equity of healthcare for all [26].
The use of questions focused on processes, policies, programs, procedures, and user
experiences has, over the course of history, emerged as an enduring and widely embraced
approach within the realm of program evaluation in healthcare [27]. Evaluation frame-
works deeply rooted in the discipline have historically served as the linchpin for assessing
the efficacy, efficiency, and quality of healthcare interventions and systems [26]. By inquir-
ing into the intricacies of processes, the robustness of policies, the efficacy of programs,
the adherence to procedures, and the experiences of users, healthcare evaluators have
systematically navigated the complex terrain of healthcare delivery, striving to unravel the
multifaceted dynamics that influence patient outcomes and system performance [27].
When evaluating the implementation of a new telehealth program for a medical
practice, it is essential to consider various aspects to ensure its success and alignment with
patient care. Here are some questions (see Table 1) to guide the evaluation process that
were developed from a review of the VA hospital telehealth policy, process, and procedure
documents for four hospitals, two Virginia and two in West Virginia in the United States.

Table 1. Organizational Alignment.

Strategic Alignment
How does the introduction of telehealth align with the overall strategic goals and mission of the medical organization?
What specific objectives or outcomes are expected from the telehealth program?
Patient Access and Engagement
How will the telehealth program improve or expand patient access to healthcare services?
What strategies are in place to promote patient engagement and participation in telehealth consultations?
Technology Infrastructure
Has the medical practice invested in the necessary technology infrastructure to support telehealth (e.g., secure video conferencing,
EHR integration)?
Are there contingency plans in case of technical issues during telehealth sessions?
Regulatory and Legal Compliance
Is the telehealth program compliant with all relevant regulations including state and federal laws (e.g., HIPAA)?
Have licensing requirements for telehealth practitioners been addressed?
Merits 2023, 3 713

Table 1. Cont.

Patient Privacy and Data Security

How are patient data protected during telehealth consultations?
Are encryption, access controls, and secure data transmission in place to safeguard patient information?
Workflow Integration
How seamlessly does telehealth integrate into existing clinical workflows?
Are there protocols for scheduling, documenting, and billing telehealth appointments?
Provider Training and Competency
Have healthcare providers received proper training in telehealth technologies and best practices?
Is there an ongoing professional development plan for telehealth competency?
Patient Education and Support
What educational materials or resources are provided to patients to prepare them for telehealth visits?
How are patients supported in using telehealth tools and navigating the process?
Quality of Care and Clinical Outcomes
How will the telehealth program measure and ensure the quality of care delivered to patients?
What key performance indicators (KPIs) or clinical outcome measures are tracked?
Cost–benefit Analysis
What is the cost-effectiveness of the telehealth program compared to traditional in-person visits?
Are there potential cost savings or revenue opportunities associated with telehealth?
Patient Feedback and Satisfaction
How is patient feedback collected and analyzed regarding their experiences with telehealth?
What improvements or adjustments have been made based on patient input?
Emergency Preparedness
Does the telehealth program have protocols in place to handle emergencies or urgent care situations effectively?
How are patients directed to in-person care when needed?
Legal and Liability Considerations
Are there legal frameworks and insurance coverage in place to address liability issues related to telehealth?
How are legal concerns and potential malpractice cases managed?
Scalability and Growth
Can the telehealth program scale to accommodate an increasing number of patients and providers?
How will the program adapt to evolving healthcare needs and technological advancements?
Community and Stakeholder Engagement
How are patients, providers, and other stakeholders involved in developing and improving the telehealth program?
Is there a communication plan for promoting the program within the community?

Evaluating these aspects comprehensively will help assess the readiness and effective-
ness of the new telehealth program ensuring that it aligns with the medical practice’s goals
and provides high-quality care to patients while meeting regulatory and legal requirements.
Evaluating the implementation of AI tools and approaches in telehealth encompasses a
multifaceted assessment essential for ensuring these technologies’ efficacy, safety, and
ethical considerations.
Several Critical Areas for Evaluation Include the Following:
Clinical Outcomes: Analyzing the impact of AI tools on patient health and outcomes
is paramount. This evaluation measures improvements in diagnosis accuracy, treatment
effectiveness, and patient recovery rates.
User Experience and Acceptance: Assessing the ease of use and satisfaction of both
healthcare providers and patients when interacting with AI-driven telehealth solutions
is crucial. User feedback and acceptance play significant roles in successfully adopting
these tools.
Merits 2023, 3 714

Data Security and Privacy: Ensuring that AI implementations comply with stringent
data protection regulations and that patient data remain secure are vital. Evaluations
should address data encryption, access control, and compliance with HIPAA or other
relevant privacy standards.
Cost-Efficiency: Analyzing the cost-effectiveness of AI-driven telehealth solutions
compared to traditional methods is essential. This level of evaluation includes assessing
reductions in healthcare costs, resource optimization, and the return on investment (ROI).
Interoperability: Evaluating the compatibility of AI tools with existing telehealth
systems and electronic health records (EHRs) is critical for seamless integration and data
exchange between healthcare providers.
Ethical Considerations: Examining the ethical implications of AI in telehealth such as
bias in algorithms, patient consent, and decision-making transparency is crucial to ensure
responsible and fair use.
Regulatory Compliance: Ensuring that AI implementations adhere to regulatory
requirements and are subject to appropriate oversight are essential. Compliance with FDA
regulations for medical devices and other relevant standards should be assessed.
Scalability and Adaptability: Evaluating the ability of AI-driven telehealth solutions
to scale and adapt to changing healthcare needs, including pandemics or shifts in patient
demographics, is vital for long-term sustainability.
Training and Education: Assessing the readiness of healthcare providers to use AI
tools effectively and providing training and educational resources to bridge any knowledge
gaps are essential for successful implementation.
Feedback Loop and Continuous Improvement: Establishing mechanisms for gathering
feedback from healthcare professionals and patients, and using this feedback to iterate and
improve AI tools and approaches are essential for ongoing success.
Legal and Liability Considerations: Evaluating liability issues and ensuring that
appropriate legal frameworks are in place is important in addressing potential malpractice
or other legal challenges related to AI use in telehealth.

13. AI Operations’ Evaluation Approaches

Comprehensive evaluation across these areas will help ensure the successful imple-
mentation of AI tools in telehealth and contribute to improved healthcare delivery and
patient outcomes. Indeed, evaluating AI implementation in telehealth requires a systematic
approach involving asking pertinent questions across various dimensions. Here are some
questions (see Table 2) to consider for a comprehensive evaluation based on the document
analysis by benchmarking procedures and evaluation telehealth technologies documents
and emerging AI telehealth documents of the two largest VA hospitals of Virginia and of
West Virginia (four total). They include the following:

Table 2. Operational Effectiveness.

Clinical Effectiveness
How has AI impacted the accuracy of diagnoses and treatment recommendations in telehealth consultations?
Can the practice provide examples of successful patient outcomes attributed to AI-driven telehealth interventions?
What clinical trials or studies have been conducted to assess the efficacy of AI in telehealth, and what were the results?
User Experience and Acceptance
How do healthcare providers and patients perceive the usability and user-friendliness of AI-powered telehealth solutions?
Have there been any issues or challenges related to user acceptance, and if so, how are they being addressed?
What feedback have users provided regarding their experiences with AI in telehealth, and how has this feedback been incorporated
into improvements?
Data Security and Privacy
How are patient data protected and secured within the AI-driven telehealth system?
What measures are in place to ensure compliance with data privacy regulations (e.g., HIPAA)?
Are there mechanisms for patients to control access to their data and provide informed consent for their use in AI applications?
Merits 2023, 3 715

Table 2. Cont.

Cost-Efficiency
How has implementing AI in telehealth affected healthcare costs and resource utilization?
Can the practice provide a cost–benefit analysis comparing AI-driven telehealth to traditional healthcare delivery methods?
Have any unexpected costs been associated with AI implementation, and if so, how were they managed?
Interoperability
How well does the AI telehealth system integrate with existing healthcare infrastructure such as electronic health records (EHRs)
and other health IT systems?
Are there any interoperability challenges that need to be addressed?
Does the system facilitate seamless data exchange between healthcare providers?
Ethical Considerations
How are bias and fairness addressed in AI algorithms used in telehealth?
What measures are in place to ensure transparency and accountability in AI decision-making processes?
Are there guidelines for disclosing AI involvement to patients during telehealth consultations?
Regulatory Compliance
Is the AI-driven telehealth solution compliant with relevant regulatory standards and certifications (e.g., FDA approval for
medical devices)?
How is compliance with data protection laws (e.g., GDPR, HIPAA) ensured?
What mechanisms are in place for handling adverse events or reporting issues related to regulatory compliance?
Scalability and Adaptability
Can the AI telehealth solution quickly scale to accommodate a growing number of patients and healthcare providers?
How adaptable is the system to changing healthcare needs such as responding to pandemics or emerging health crises?
Training and Education
What training and educational resources are available to healthcare providers to use AI tools in telehealth effectively?
How are healthcare professionals kept up-to-date with AI advancements and best practices?
Are there certification programs for AI proficiency in telehealth?
Feedback and Continuous Improvement
How is feedback collected from healthcare providers and patients regarding their experiences with AI in telehealth?
How are suggestions and concerns addressed and used for continuous improvement?
Is there a structured process for iteratively enhancing AI implementations?
Legal and Liability Considerations
What legal frameworks and protocols are in place to handle liability issues related to AI in telehealth?
How are potential malpractice or legal challenges addressed, and is there insurance coverage for AI-related issues?

Evaluating Artificial Intelligence (AI)-Driven Telehealth Technologies

Evaluating the utility and effectiveness of AI-driven telehealth technologies is crucial
to ensure these innovations meet their intended goals and provide high-quality healthcare
services. Here are some critical areas for evaluation and questions to be asked in the
assessment process (see Table 3).

Table 3. Operational Process Effectiveness.

AI Operations’ Evaluation Approaches Clinical Effectiveness

How has AI impacted the accuracy of diagnoses and treatment
Comprehensive evaluation across various dimensions
recommendations in telehealth consultations?
Can the practice provide examples of successful patient
outcomes attributed to AI-driven telehealth interventions?
What clinical trials or studies have been conducted to assess the
efficacy of AI in telehealth, and what were the results?
Merits 2023, 3 716

Table 3. Cont.

User Experience and Acceptance Data Security and Privacy

How do healthcare providers and patients perceive the usability How are patient data protected and secured within the
and user-friendliness of AI-powered telehealth solutions? AI-driven telehealth system?
Have there been any issues or challenges related to user What measures are in place to ensure compliance with data
acceptance, and if so, how are they being addressed? privacy regulations (e.g., HIPAA)?
What feedback have users provided regarding their experiences
Are there mechanisms for patients to control access to their data
with AI in telehealth, and how has this feedback been
and provide informed consent for their use in AI applications?
incorporated into improvements?
Cost-Efficiency Interoperability
How well does the AI telehealth system integrate with existing
How has implementing AI in telehealth affected healthcare
healthcare infrastructure such as electronic health records
costs and resource utilization?
(EHRs) and other health IT systems?
Can the practice provide a cost–benefit analysis comparing Are there any interoperability challenges that need to
AI-driven telehealth to traditional healthcare delivery methods? be addressed?
Have any unexpected costs been associated with AI Does the system facilitate seamless data exchange between
implementation, and if so, how were they managed? healthcare providers?
Ethical Considerations Regulatory Compliance
Is the AI-driven telehealth solution compliant with relevant
How are bias and fairness addressed in AI algorithms used
regulatory standards and certifications (e.g., FDA approval for
in telehealth?
medical devices)?
What measures are in place to ensure transparency and How is compliance with data protection laws (e.g., GDPR,
accountability in AI decision-making processes? HIPAA) ensured?
Are there guidelines for disclosing AI involvement to patients What mechanisms are in place for handling adverse events or
during telehealth consultations? reporting issues related to regulatory compliance?
Scalability and Adaptability Training and Education
Can the AI telehealth solution quickly scale to accommodate a What training and educational resources are available to
growing number of patients and healthcare providers? healthcare providers to use AI tools in telehealth effectively?
How adaptable is the system to changing healthcare needs such How are healthcare professionals kept up-to-date with AI
as responding to pandemics or emerging health crises? advancements and best practices?
Are there certification programs for AI proficiency in telehealth?
Feedback and Continuous Improvement Legal and Liability Considerations
How is feedback collected from healthcare providers and What legal frameworks and protocols are in place to handle
patients regarding their experiences with AI in telehealth? liability issues related to AI in telehealth?
How are suggestions and concerns addressed and used for How are potential malpractice or legal challenges addressed,
continuous improvement? and is there insurance coverage for AI-related issues?
Evaluating Artificial Intelligence (AI)-Driven Telehealth
Clinical Outcomes
Technologies
Evaluating the utility and effectiveness of AI-driven Are there measurable improvements in patient health outcomes
telehealth technologies using AI-driven telehealth technologies?
Here are some critical areas for evaluation and questions to be How do these outcomes compare to traditional in-person care
asked in the assessment process or other telehealth modalities?
What clinical conditions or specialties benefit the most from
AI-driven telehealth, and where are the limitations?
Merits 2023, 3 717

Table 3. Cont.

Patient Satisfaction Accessibility and Reach

What is the level of patient satisfaction with AI-driven To what extent does AI-driven telehealth improve access to
telehealth services? healthcare, particularly in underserved or remote areas?
Do patients feel that their healthcare needs are adequately Are there barriers to access such as technological challenges or
addressed through this technology? lack of internet connectivity?
Are there any disparities in satisfaction among different
How does telehealth affect healthcare disparities?
patient demographics?
Cost-effectiveness Diagnostic Accuracy
What are the cost implications of implementing AI-driven How accurate are the AI algorithms in diagnosing medical
telehealth technologies for healthcare providers and patients? conditions or predicting patient outcomes?
Does telehealth reduce healthcare costs such as travel expenses Are there instances of misdiagnosis or over-reliance on
and hospital readmissions? AI recommendations?
Are there potential cost savings for healthcare systems What measures are in place to ensure the ongoing improvement
and payers? of AI algorithms?
Privacy and Security Provider Experience
Are patients’ health data adequately protected, and do
How do healthcare professionals perceive the integration of
AI-driven telehealth technologies comply with relevant privacy
AI-driven telehealth into their practice?
regulations (e.g., HIPAA)?
What security measures are in place to prevent data breaches Does telehealth enhance or detract from their workflow, and are
or cyberattacks? they satisfied with the technology?
Do patients have concerns about the privacy of their Are there opportunities for training and support to improve
health information? provider comfort and proficiency with these tools?
Integration with Existing Systems Regulatory Compliance
How seamlessly do AI-driven telehealth technologies integrate
Are AI-driven telehealth technologies compliant with local and
with existing electronic health records (EHR) and healthcare IT
national regulations governing telehealth and medical practice?
infrastructure?
Are there interoperability challenges, and if so, how are they Do they meet the standards set by relevant medical boards
being addressed? and organizations?
Ethical Considerations Long-Term Impact
Are there ethical dilemmas associated with AI-driven telehealth, What is the potential long-term impact of AI-driven telehealth
such as using patient data for research or concerns about on healthcare delivery, patient outcomes, and the
algorithm bias? healthcare workforce?
How will these technologies evolve in response to emerging
How are these ethical concerns being addressed in practice?
healthcare needs and challenges?
Are there mechanisms for ongoing community engagement to Evaluating AI-driven telehealth technologies requires a
ensure that the technology aligns with the populations and the multifaceted approach considering clinical, technical, ethical,
community’s needs and preferences? and societal aspects.
What steps are being taken to ensure that historically Comprehensive assessments can help ensure that these
underserved populations are included in the development and technologies enhance the quality, accessibility, and efficiency of
testing of AI-driven telehealth technologies? healthcare while addressing potential challenges and concerns.
How is health information communicated through AI-driven Asking these questions and conducting a thorough evaluation
telehealth tools, and is it presented in a way that is can help assess the impact and effectiveness of AI
understandable and accessible to individuals with varying implementation in telehealth while ensuring ethical, legal, and
levels of health literacy? regulatory compliance.
Merits 2023, 3 718

Table 3. Cont.

Ethical Considerations Long-Term Impact

How does the deployment of AI-driven telehealth account for
geographic disparities, especially in rural or remote areas with
limited access to traditional healthcare services?
Are there provisions for language access, considering linguistic
diversity within the community, and how are language-related
health disparities being addressed?
Are there plans to provide training or support for individuals
who may face technological barriers in using telehealth services?
How will the deployment of AI-driven telehealth tools impact
access to healthcare services in different communities, especially
those facing economic challenges?
Are there potential disparities in the affordability of devices or
internet connectivity required for telehealth, and how can these
be addressed?

14. Conclusions
The seismic shift triggered by the COVID-19 pandemic has thrust the healthcare sector
into a new era defined by the integration of telehealth and AI-driven telemedicine technologies.
The promises of enhanced accessibility, efficiency, and responsiveness to patient needs are
captivating, but as these technologies become integral components of healthcare systems
globally, the need for systematic evaluation processes becomes increasingly evident.
This exploration has journeyed through the transformative wave, highlighting a
crucial gap in research that addresses the practical needs and perspectives of those deeply
immersed in healthcare operations and the broader world of practice. While the benefits
of telehealth and AI-driven telemedicine technologies rightly capture attention, our focus
sharpens on the critical need for systematic evaluation, especially within the complexities
of real-world healthcare settings.
This paper has endeavored to emphasize the paramount importance of developing
rigorous evaluation frameworks tailored to the unique challenges of the evolving landscape.
The accelerated deployment of these technologies, fueled by the urgency of the crisis,
underscores the necessity for standardized evaluation processes. The absence of such
frameworks poses multifaceted challenges, from uncertainties about long-term efficacy to
concerns regarding patient safety, data security, and ethical considerations.
In response to these challenges, our exploration has advocated for the establishment
of adaptable, multidimensional evaluation mechanisms aligned with clinical objectives and
regulatory standards. Beyond the theoretical promises of these technologies, their responsi-
ble and effective integration into healthcare systems demands a nuanced understanding of
their impact on both patient outcomes and healthcare operations.
As we navigate this discourse, our call to action resonates; the development of evalu-
ation processes empower stakeholders to optimize the deployment of telehealth and AI
technologies. However, the true realization of these technologies’ potential lies not merely
in theoretical discussions but also in practical tools, checklists, and real-world approaches
to the evaluation of operations and outcomes. This is where the transformative power of
telehealth and AI-driven telemedicine will find its tangible expression.
In the wake of the COVID-19 pandemic, the rapid and widespread integration of
telehealth technologies and artificial intelligence (AI)-driven telemedicine technologies into
healthcare systems have brought both unprecedented promise and unforeseen challenges
to the forefront of our healthcare landscape. These innovations hold the potential to
reshape healthcare delivery and to render it more accessible, efficient, and responsive to
the ever-evolving needs of patients and providers. However, this transformative journey
Merits 2023, 3 719

has accentuated the critical need for systematic and rigorous evaluation processes that can
comprehensively scrutinize the operations and outcomes of these technologies.
The absence of standardized evaluation mechanisms has cast a shadow over the path
ahead, raising multifaceted concerns ranging from uncertainties about long-term efficacy
to questions of patient safety, data security, and ethical considerations. Yet, amid these chal-
lenges, a clear imperative emerges: the development of adaptable and multidimensional
evaluation frameworks that are harmoniously attuned to the dynamic nature of telehealth
and AI technologies. These frameworks must not only align with clinical objectives but
also stand in compliance with the evolving regulatory landscape.
As healthcare organizations stand at the threshold of a new era in healthcare charac-
terized by the transformative power of telehealth and AI-driven telemedicine, our commit-
ment to establishing robust evaluation processes assumes paramount importance. These
processes are not just a formality; they are the linchpin that enables us to optimize the
deployment of telehealth and AI technologies, fostering a healthcare ecosystem firmly
rooted in patient-centered care while vigilantly addressing emerging challenges. In this era
of healthcare transformation, the establishment of comprehensive and adaptable evaluation
frameworks emerges as a resounding call to action. It is an essential endeavor that holds
the key to unlocking the full potential of telehealth and AI-driven telemedicine in the
post-COVID-19 healthcare landscape.

15. Recommendations for Future Research

The evolving landscape of telehealth and AI-driven telemedicine technologies as
catalyzed by the COVID-19 pandemic invites a rich array of opportunities for future
research endeavors. To advance our understanding and harness the full potential of these
transformative innovations, the following avenues for research are proposed:
1. Ethnographic studies: Employ ethnographic approaches to immerse researchers within
healthcare settings where telehealth and AI-driven telemedicine are implemented.
Observe daily practices, interactions, and challenges to gain a nuanced understanding
of the technologies’ impact.
2. Stakeholder perspectives: Investigate the perspectives of key stakeholders, including
healthcare administrators, policymakers, and technology developers. Qualitatively
assess their motivations, concerns, and visions for the future of telehealth and AI-
driven telemedicine.
3. Case studies: Undertake in-depth case studies of healthcare organizations that have
successfully integrated telehealth and AI-driven telemedicine. Examine the contextual
factors, strategies, and challenges encountered in their implementation journeys.
4. Cultural and societal impact: Investigate the cultural and societal impact of tele-
health and AI in healthcare. Qualitatively explore how these technologies influence
healthcare disparities, patient–provider relationships, and cultural norms surrounding
healthcare practices.
5. Ethical framework development: Given the ethical complexities inherent in AI-driven
healthcare, research should focus on developing robust ethical frameworks that ad-
dress issues such as informed consent, data privacy, and algorithmic bias. Additionally,
investigations into the ethical implications of AI’s role in clinical decision-making
are paramount.
6. Data security and privacy: Research should explore cutting-edge data security and
privacy measures to safeguard patient information in telehealth and AI systems.
Assessments of potential vulnerabilities, strategies for encryption, and compliance
with evolving data protection regulations are essential.
7. User experience and acceptance: In-depth studies on the user experience of patients,
healthcare providers, and support staff when using telehealth and AI-driven tech-
nologies are imperative. Insights into the user’s acceptance, usability, and factors
influencing adoption will be instrumental in refining these systems.
Merits 2023, 3 720

8. Healthcare disparities and equity: Investigating the role of telehealth and AI in address-
ing or exacerbating healthcare disparities is vital. Research should identify barriers to
evaluate interventions to reduce disparities and assess the impact of these technologies
on marginalized populations.
9. AI algorithm improvement: Continuous research into enhancing AI algorithms is
essential. This research includes refining diagnostic accuracy, reducing bias, and
optimizing predictive capabilities. Comparative studies on various AI models and
their performance are warranted.
10. Regulatory and policy frameworks: Research should explore the development of
adaptable regulatory and policy frameworks that align with the dynamic nature of
telehealth and AI in healthcare. Evaluations of the impact of regulatory changes on
technology adoption and patient care are necessary.
11. Interoperability and Integration: Investigations into improving the interoperability
of telehealth and AI systems with existing healthcare infrastructure are needed. Re-
search should identify best practices for seamless integration and data exchange
across platforms.
In conclusion, the ongoing transformation of healthcare through telehealth and AI-driven
telemedicine necessitates a multifaceted research agenda. By addressing these critical research
areas, we can maximize the benefits of these innovations, ensure their ethical and secure
use, and advance patient-centered care in a rapidly evolving healthcare landscape. The
collaborative efforts of researchers, policymakers, and healthcare practitioners are pivotal in
shaping the future of healthcare delivery in the post-COVID-19 era.

Funding: This research received no external funding.

Institutional Review Board Statement: Not applicable.
Informed Consent Statement: Not applicable.
Data Availability Statement: Not applicable.
Conflicts of Interest: The author declares no conflict of interest.

References
1. El-Yafouri, R. Unlocking the Future of Health Care: The Power of Digital Transformation and AI. Med. Econ. 2023. Available online:
https://ramaonhealthcare.com/unlocking-the-future-of-health-care-the-power-of-digital-transformation-and-ai/ (accessed on
17 July 2023).
2. Karppinen, K.; Moe, H. What We Talk about When We Talk about Document Analysis. In Trends in Communication Policy Research:
New Theories, Methods and Subjects; University of Chicago: Chicago, IL, USA, 2012; pp. 177–193.
3. Bowen, G.A. Document Analysis as a Qualitative Research Method. Qual. R. J. 2009, 9, 27–40. [CrossRef]
4. Brown, M.; Hale, K. Applied Research Methods in Public and Nonprofit Organizations; John Wiley & Sons: Hoboken, NJ, USA, 2014.
5. Hedrick, T.E.; Bickman, L.; Rog, D.J. Applied Research Design: A Practical Guide; Sage: Newbury Park, CA, USA, 1993.
6. Hulland, J. Conceptual Review Papers: Revisiting Existing Research to Develop and Refine Theory. AMS Rev. 2020, 10, 27–35.
[CrossRef]
7. Donaldson, S.I.; Christie, C.A.; Mark, M.M. What Counts as Credible Evidence in Applied Research and Evaluation Practice? Sage:
Newbury Park, CA, USA, 2009.
8. Dalglish, S.L.; Khalid, H.; McMahon, S.A. Document Analysis in Health Policy Research: The READ Approach. Health Policy Plan.
2020, 35, 1424–1431. [CrossRef] [PubMed]
9. Bailey, J.E.; Gurgol, C.; Pan, E.; Njie, S.; Emmett, S.; Gatwood, J.; Gauthier, L.; Rosas, L.G.; Kearney, S.M.; Robler, S.K.; et al. Early
Patient-Centered Outcomes Research Experience Using Telehealth to Address Disparities: Scoping Review. J. Med. Int. R. 2021,
23, e28503.
10. Amann, J.; Blasimme, A.; Vayena, E.; Frey, D.; Madai, V.I. Explainability for Artificial Intelligence in Healthcare: A Multidisci-
plinary Perspective. BMC Med. Inf. Decis. Mak. 2020, 20, 310. [CrossRef] [PubMed]
11. Bohr, A.; Memarzadeh, K. (Eds.) Artificial Intelligence in Healthcare; Academic Press: Cambridge, MA, USA, 2020.
12. Machmud, M.; Chairun Nasirin, N.; Salahudin, S.; Tawakkal, B. Artificial Intelligence in the Public Health Sector: The Use of
Telemedicine in Indonesia during COVID-19. Palarch’s J. Archaeol. Egypt/Egyptol. 2020, 17, 10106–10118.
13. Manne, R.; Kantheti, S.C. Application of Artificial Intelligence in Healthcare: Chances and Challenges. Curr. J. Appl. Sci. Technol.
2021, 40, 78–89. [CrossRef]
Merits 2023, 3 721

14. Subbhuraam, V.; Panigrahi, D. Telemedicine. In Predictive Analytics in Healthcare, Volume 1: Transforming the Future of Medicine; IOP
Publishing: Bristol, UK, 2021; pp. 1-1–1-15.
15. Kadu, A.; Singh, M. Comparative Analysis of E-Health Care Telemedicine System Based on Internet of Medical Things and
Artificial Intelligence. In Proceedings of the 2021 2nd International Conference on Smart Electronics and Communication
(ICOSEC), Trichy, India, 7–9 October 2021; pp. 1768–1775.
16. Yu, H.; Zhou, Z. Optimization of IoT-Based Artificial Intelligence-Assisted Telemedicine Health Analysis System. IEEE Access
2021, 9, 85034–85048. [CrossRef]
17. Andrikopoulou, E. The Rise of AI in Telehealth. In Emerging Practices in Telehealth; Academic Press: Cambridge, MA, USA, 2023;
pp. 183–207.
18. Ellahham, S. Artificial Intelligence: The Future for Diabetes Care. Am. J. Med. 2020, 133, 895–900. [CrossRef] [PubMed]
19. Seetharam, K.; Kagiyama, N.; Sengupta, P.P. Application of Mobile Health, Telemedicine and Artificial Intelligence to Echocardio-
graphy. Echo Res. Prac. 2019, 6, R41–R52. [CrossRef] [PubMed]
20. George, A. An Inquiry into the Lived Experience of Tele-ICU Nurses’ Practice. Ph.D. Dissertation, Adelphi University, Garden
City, NY, USA, 2020.
21. Sidney, H.; Sitarah, M.; Dileep, R.; Sanu, A.; Ryan, B.; Carl, B. Shared Features of Successful Tele-ICU Models—A Narrative
Review of Successful Implementation with a Focus on LMIC Models. Health Policy Technol. 2023, 12, 100802.
22. Dumitrascu, O.M.; Demaerschalk, B.M. Telestroke. Curr. Cardiol. Rep. 2017, 19, 85. [CrossRef] [PubMed]
23. Solenski, N.J. Telestroke. Neuroimaging Clin. 2018, 28, 551–563. [CrossRef] [PubMed]
24. Havenga, E.; Swanepoel, D.W.; Le Roux, T.; Schmid, B. Tele-Intervention for Children with Hearing Loss: A Comparative Pilot
Study. J. Telemed. Telecare 2017, 23, 116–125. [CrossRef] [PubMed]
25. Noel, K.; Fabus, R. Telehealth: Incorporating Interprofessional Practice for Healthcare Professionals in the 21st Century; Elsevier:
Amsterdam, The Netherlands, 2022.
26. Nepal, S.; Li, J.; Jang-Jaccard, J.; Alem, L. A Framework for Telehealth Program Evaluation. Telemed. e-Health 2014, 20, 393–404.
[CrossRef] [PubMed]
27. Grembowski, D. The Practice of Health Program Evaluation; Sage: Newbury Park, CA, USA, 2015.

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual
author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to
people or property resulting from any ideas, methods, instructions or products referred to in the content.