U.S. flag

An official website of the United States government

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Hatef E, Wilson RF, Hannum SM, et al. Use of Telehealth During the COVID-19 Era [Internet]. Rockville (MD): Agency for Healthcare Research and Quality (US); 2023 Jan.

Cover of Use of Telehealth During the COVID-19 Era

Use of Telehealth During the COVID-19 Era [Internet].

Show details

2Methods

2.1. Review Approach

We conducted a mixed-methods review that considered both quantitative and qualitative studies.20 The topic of this report was developed by a Learning Health System representative in consultation with the Agency for Healthcare Research and Quality (AHRQ) (https://effectivehealthcare.ahrq.gov/products/learning-health-systems-panel/overview). Initially, a panel of stakeholders gave input on the Key Questions (KQs) to be examined; these KQs were posted on AHRQ’s website for public comment in June 2021 for 3 weeks and revised in response to comments. A panel of technical experts provided high-level content and methodological expertise throughout the development of the review protocol, including of the analytic figure seen in Figure 1. The final protocol is posted on the AHRQ website at https://effectivehealthcare.ahrq.gov/products/virtual-health-covid/protocol.

2.2. Key Questions

KQ 1.

What are the characteristics of patients, providers, and health systems using telehealth during the COVID-19 era? Specifically:

  1. What are the characteristics of patients (e.g., age, race/ethnicity, gender, socioeconomic status, education, geographic location [urban vs. rural])?
  2. What are the characteristics of providers and health systems (e.g., specialty, geographic location, private practice, hospital-based practice)?
  3. How do the characteristics of patients, providers, and health systems differ between the first 4 months of the COVID-19 era versus the remainder of the COVID-19 era?

KQ 2.

What are the benefits and harms of telehealth during the COVID-19 era?

  1. Does this vary by type of telehealth intervention (e.g., telephone, video visits)?
  2. Does this vary by patient characteristics (e.g., age, gender, race/ethnicity, type of clinical condition or health concern, geographic location)?
  3. Does this vary by provider and health system characteristics (e.g., specialty, geographic location, private practice, hospital-based practice)?

KQ 3.

What is considered a successful telehealth intervention, and what are the barriers and facilitators of these interventions during the COVID-19 era:

  1. From the patient or caregiver perspective?
  2. From the provider perspective?

KQ 4.

What strategies have been used to implement telehealth interventions during the COVID-19 era?

This figure depicts the Key Questions within the context of PICOTS (population, intervention, comparator, outcomes, timing, and setting). In general, the figure illustrates how telehealth may result in intermediate outcomes such as patient satisfaction, patient-centered service, ease of access and use, communication challenges, provider satisfaction, provider ease of use, and/or final health outcomes such as clinical outcomes, maintenance of utilization, improved quality measures, service substitution versus duplication, and maintenance of utilization. Also, adverse events may occur at any point after patients receive the treatment.

Figure 1

Analytic framework for telehealth during COVID-19. ED = emergency department; IP = inpatient; KQ = Key Question; OP = outpatient

2.3. Study Selection

We conducted a search for studies of any design about telehealth that were conducted after the onset of the era of COVID-19. We searched PubMed®, CINAHL®, PsycINFO®, and the Cochrane Central Register of Controlled Trials (see Appendix A Methods, Table A.1.1 through A1.3). An information specialist reviewed search strategies using the Peer Review of Electronic Search Strategies (PRESS) guidelines.21 Searches were conducted in July 2021, and updated in May 2022, and were limited to studies published during the era of COVID-19 (beginning March 2020). Additional studies were identified from reference lists and experts. Searches will be updated while the draft report is under review. A Supplemental Evidence and Data for Systematic review (SEADS) portal was posted in November 2021 and a Federal Register Notice was posted in October 2021 for this review.

We included studies according to a PICOTS (population, intervention, comparators, outcomes, timing, setting) framework (see Appendix A Methods, Table A.2). For all KQs, we included patients of any age (and their caregivers), all centers/locations of patient care, and healthcare providers of any type. We included only remotely delivered, synchronous medical services (e.g., telephone/audio, video visits) between a patient and a healthcare provider in an ambulatory setting or emergency department (ED) providing acute/urgent care, routine/chronic care, mental health services, wellness visits, post-hospital discharge care, and patient and specialist communications facilitated by an ED physician in an ED. Asynchronous and automated (artificial intelligence or semi-automated applications) services were not included. For KQ4, we included studies of implementation strategies for telehealth. All studies took place during the COVID-19 era (starting March 2020). We defined telehealth as any healthcare provided outside of a medical office via phone or video and healthcare provided in an ED by a specialist via phone or video, limiting it to the interaction between a patient or their caregiver and a healthcare provider. We included all outpatient populations in countries with a population similar to that of the United States, using the Organisation for Economic Cooperation and Development (OECD) nations excluding those with a World Health Organization classification below “upper income”22 (Appendix A, Methods Table A.3). Applicable study designs and synthesis methods are listed in Appendix A, Methods Table A.4.

We managed and screened the results of the search using DistillerSR (Evidence Partners, 2010). Citations were screened at abstract and full-text level by two screeners, independently. At both levels, exclusion required that both screeners agreed. Differences between reviewers regarding abstract or full-text eligibility were resolved through consensus. Owing to the volume of new studies identified during the update, we used the artificial intelligence (AI) feature in DistillerSR during abstract screening to prioritize studies most eligible for inclusion. A threshold of 95 percent was used: when the AI detected that 95 percent of potentially eligible studies were detected, we discontinued screening.

2.4. Data Extraction

For all KQs, study and participant characteristics were extracted by one reviewer, and a second reviewer confirmed accuracy and completeness. KQ-specific information was extracted in the same manner for KQs 1, 2, and 4. Information specific to KQ3 (qualitative data and survey data) was extracted by one reviewer, and then reviewed and compiled by a second reviewer.

In cases where the study period began prior to the COVID-19 era, we extracted data in the following manner:

  • If data collection began between 1 January and 11 March 2020 and was in response to the COVID-19 pandemic, we abstracted all data.
  • If data collection began prior to the era of COVID-19 and extended into the era of COVID-19, we extracted data for the COVID-19 era. If it was not possible to extract separate data, the study was excluded.

If data were presented for countries with populations similar to the United States, according to the OECD,22 and countries with populations unlike the United States, we only extracted data from countries with populations similar to the United States.

2.5. Risk of Bias Assessment

For KQ2 and KQ4, paired investigators independently assessed studies included in the syntheses for risk of bias. We used the Cochrane Risk of Bias Tool, Version 2, for assessing the risk of bias of randomized controlled trials23 and the Cochrane Risk of Bias Assessment Tool for Non-Randomized Studies of Interventions (ROBINS-I) tool24 for non-randomized trials.

For qualitative studies (KQ3), reviewers independently assessed study quality using the Critical Appraisals Skills Programme (CASP) tool.25 The CASP criteria includes 10 elements. We included eight of the 10 elements in our assessment (excluding both the relationship between researcher and participants, as this was rated “no” across all studies, and value of the research, as we did not consider this to be an appropriate question for an emerging body of evidence). We followed Long et al.26 while assessing each criterion: yes, somewhat/to some extent, can’t tell, and no. Each included element was scored equally following Njau et al.27: 2, criterion is completely met (yes); 1, criterion is partially met (somewhat/to some extent); 0, criterion was not met or not mentioned (can’t tell, or no). A maximum score of 16=high quality; 12.8-15.9=moderate quality; 9.6-12.79=low quality; <9.6=very low quality. We did not assess risk of bias for studies selected for KQ1 or for surveys (KQ3).

2.6. Data Synthesis and Analysis

2.6.1. Key Question 1

As this was a descriptive question, we did not conduct a systematic review; we selected a subset of studies to describe; we did not conduct risk of bias of those studies, and did not conduct strength of evidence grading. We aggregated information and presented descriptive statistics on the characteristics of users (by patients, providers, and health systems) during the COVID-19 era. In the appendix, we listed all studies identified; but, in the results, we described a subset of studies that were large (greater than one million patients), with comprehensive assessment of the patient or provider or telehealth characteristics, and with a sampling method providing results that are representative of the U.S. population.

2.6.2. Key Question 2

We addressed the question of benefits and harms of telehealth during the COVID-19 era through a synthesis of studies that provided comparative data for outcomes. Studies without comparative data or design are described briefly in the report but otherwise are listed in the appendix. We were unable to conduct meta-analysis owing to limited and heterogeneous data, missing information, and variation in the outcomes measured. We considered an effect or difference clinically meaningful if it would result in a change in the clinical practice or care plan for the patient.

2.6.3. Key Question 3

We addressed the question of what is considered a “successful” telehealth intervention by evaluating satisfaction/dissatisfaction and barriers/facilitators through a qualitative evidence synthesis. We created a matrix of users (i.e., patients, caregivers, providers), their characteristics, and their perspectives or expectations (themes) of a satisfactory telehealth service. We followed the same approach to synthesize patient, caregiver, or provider perspectives on the barriers or facilitators to telehealth. The themes were extracted to saturation. The findings of each research paper are “data” points and we evaluated these data to determine themes. When no new findings emerged, we considered thematic saturation to have been achieved. For the results from the updated search, we synthesized only those studies which added new theme(s), provided more evidence for a theme(s), and addressed the later COVID-19 period (i.e., where thematic saturation had not been achieved). We did not add survey data during the updated search.

Survey data on patient, caregiver, or provider perceptions of satisfaction, barriers, and facilitators of telehealth were not included in the qualitative evidence synthesis. Survey results are discussed as supporting or not supporting the findings of the qualitative evidence synthesis. The surveys were also mapped to the above matrix. We used a convergent segregated approach to synthesis and integration of the quantitative and qualitative data.28 In this approach, the syntheses of qualitative and quantitative (survey) studies are conducted separately, and then these results are juxtaposed to determine how the findings complement each other. Taking this approach allowed us to identify how the data from quantitative and qualitative sources complement one other (converge) and also identify where gaps between the two bodies of literature exist.

2.6.4. Key Question 4

We synthesized the strategies and outcomes presented in the studies using an adaptation of the implementation outcomes and categories defined by Proctor et al. (2011).29

2.7. Grading the Strength of the Body of Evidence

We graded the body of evidence separately for quantitative and qualitative studies. For the systematic review of quantitative studies for KQs 2 and 4, we used the grading scheme recommended in the AHRQ Methods Guide for Effectiveness and Comparative Effectiveness Reviews (Methods Guide).30 For studies included in the qualitative evidence syntheses in KQ3, we followed the GRADE-CERQual (Confidence in the Evidence from Reviews of Qualitative research) approach.31–37 In each case, two reviewers independently conducted the grading with input from other team members, as needed, to reach consensus. Grading was not completed for KQ1.

2.8. Peer Review and Public Commentary

For a period of 3 weeks, experts in healthcare (providers), telehealth (industry, policy, implementation, and research), and qualitative synthesis, along with patient and caregiver advocates and federal representatives, were asked to provide external peer review of this report; AHRQ and an associate editor also provided comments. Following peer review, the revised draft report was posted on the AHRQ website for 4 weeks to elicit public comment. We addressed all reviewer comments, revising the report as appropriate. A disposition of comments table including comments from public reviewers will be posted on the AHRQ website 3 months after the Agency posts the final report.