Literature Search Strategy

We conducted electronic searches in Ovid MEDLINE^®, CINAHL^®, Embase^®, and Cochrane CENTRAL from July 1, 2016, to March 4, 2022. (See Appendix A for full strategies). This captures studies of systems that rely on more current technology and follows searches from a recent report that provided an evidence map of telehealth services for women (search end date was December 2016).⁴⁹ We reviewed the studies included in the evidence map for consideration in this review and included information on the dates the studies were conducted, the technologies used, and the dates of publication. Reference lists of included systematic reviews were screened for additional studies and relevant references were carried forward. A Federal Register Notice was posted to encourage submission of unpublished studies through a Supplemental Evidence and Data for Systematic review (SEADS) portal.

Inclusion and Exclusion Criteria and Study Selection

Criteria were established a priori to determine eligibility for inclusion and exclusion of abstracts in accordance with the AHRQ Methods Guide⁴⁸ and search strategies were peer reviewed. Study eligibility criteria for this CER were based on the population, intervention, comparisons, outcomes, settings, and study designs of interest (PICOS) framework and the Key Questions. The population of interest was adolescent and adult women (≥13 years old), including those who are pregnant, and those eligible for screening, counseling, or treatment for reproductive health services (family planning, contraception, and sexually transmitted infection [STI] counseling) and interpersonal violence (IPV). Details regarding the PICOS are summarized in Table 1 with additional details in Appendix Table A-1. Specific outcomes for each preventive service considered are described in detail in Appendix Table A-2.

For this review, the term women is used in a biological context, where applicable (e.g., individuals with potential for becoming pregnant without contraception), and can be applied to individuals of all gender identities, including cisgender, transgender, gender non-binary, or otherwise gender expansive for relevant services.

Reproductive health services considered for this review include family planning, contraception, and STI counseling. For this review, family planning services were defined based on Title X guidelines⁵⁰ and include preconception counseling and birth spacing; contraceptive care (screening, counseling, provision, and followup care) was considered separately under reproductive health services. We considered contraceptive care that could be delivered via telehealth by a broad range of health care workers (e.g., physicians, nurses, pharmacists, counselors). Telehealth services for IPV include screening, diagnosis, and treatment for intimate partner violence and domestic violence.

The term telehealth is used to define services that may include the use of information and telecommunications technology in healthcare delivery for a specific patient involving a clinician across distance or time, such as remote real-time clinical visits and remote monitoring. For this review we refer to telehealth when considering interventions that use technology to facilitate interactions at a distance between specific patients and clinicians and are bidirectional or link to clinical care. Interactions could occur over time (asynchronous) as well as over distance. We considered telephone conversations, e-mail, and short message service (SMS) texts to be telehealth if they allow interaction between patient and clinician (bidirectional) and could replace or supplement an in-person interaction. Interventions were not included if they occurred only in one direction or if they were not personalized (e.g., phone, email or text message notifications, generic messages sent to a group of patients). For example, an app that collects data but does not involve clinical decision-making or individualized patient care was not eligible for inclusion, but an app or website that is bidirectional and personalized based on specific patient input was considered.

Study designs considered for inclusion were comparative studies of any design including trials and observational studies. We considered observational cohort studies, pre-post designs (i.e., comparison of the same population across time points), and before-after studies (i.e., comparison of two time points; may not have the same population). Qualitative studies that evaluated patient and clinician preferences, and barriers to and facilitators of telehealth were included. Descriptive studies with no outcome data or studies that included only data from one point in time (cross-sectional) were not included, although they were considered for studies evaluating the impact of the COVID-19 pandemic and for the Contextual Question. Also excluded were modeling studies or studies that used synthetic data. We reviewed existing systematic reviews and included their results if appropriate. References lists of systematic reviews were also used to identify relevant studies. Commentaries, letters, and articles that described telehealth systems or implementation strategies but did not assess impact were excluded, as were studies published only as conference abstracts. Inclusion was restricted to English-language articles, and studies of nonhuman subjects were excluded. Studies had to report original data to be included.

To ensure accuracy, all excluded abstracts were dual reviewed by two investigators. Each full-text article was independently reviewed for eligibility by two team members. All disagreements were resolved through a consensus process between investigators.

Table 1

PICOS—inclusion and exclusion criteria.

Data Abstraction and Data Management

Dual review of abstracts was conducted using prespecified inclusion criteria and DistillerSR software. Discrepancies were resolved by discussion and consensus. Investigators tracked results in an EndNote database (Thomson Reuters). For studies meeting inclusion criteria, evidence tables were constructed with the following data: study design, year, setting, country, sample size, patient and clinician type and characteristics (e.g., age, sex, race, reason for presentation, diagnosis, clinician training/background/scope of practice and primary care or specialty type), intervention characteristics (e.g., mode of delivery, duration or frequency, function), and results relevant to each Key Question (KQ), as outlined in the previous PICOS section. All study data were verified for accuracy and completeness by a second team member.

Risk of Bias Assessment of Individual Studies

Predefined criteria were used to assess the risk of bias (also referred to as quality or internal validity) for each individual included study, using criteria appropriate for the study designs (Appendix A). Controlled trials and observational studies were assessed using a priori established criteria consistent with the AHRQ EPC approach recommended in the chapter, Assessing the Risk of Bias of Individual Studies, described in the Methods Guide for Effectiveness and Comparative Effectiveness Reviews⁴⁸ (Appendix A). Randomized controlled trials (RCTs) were evaluated using criteria and methods developed by the Cochrane Back and Neck Group,⁵¹ cohort and other observational studies of interventions were evaluated using criteria developed by the U.S. Preventive Services Task Force,⁵² and followed the approach recommended in the AHRQ Methods Guide chapter “Assessing the Risk of Bias of Individual Studies When Comparing Medical Interventions.”⁴⁸ For RCTs, we focused on randomization, allocation concealment, analysis according to randomized groups (intention-to-treat analysis), and attrition. Cohort studies were included to fill gaps in evidence for studies not specifically addressing the COVID-19 pandemic. For before-after studies and interrupted time-series studies assessing effects during the COVID-19 pandemic, criteria included prespecified outcome measures, enrollment methods, and controlling of temporal trends, derived from a National Institutes of Health checklist.⁵³ For surveys, criteria were derived from a set of questions developed by members of this review team for a Health Information Exchange systematic review⁵⁴ and evaluated reported response rates; sampling strategy, selection, and sample characteristics; survey questions; and consideration of confounders and analyses. (See Appendix A.)

Each study evaluated was independently reviewed for risk of bias by two team members. Any disagreements were resolved through consensus. Based on the risk of bias assessment, individual included studies were rated as “low,” “moderate,” or “high” risk of bias. High risk of bias studies were not excluded a priori but were considered to be less reliable than low or moderate risk of bias studies when synthesizing the evidence.

Data Analysis and Synthesis

Evidence tables identify study characteristics, results of interest, and risk of bias ratings for all included studies and summary tables highlight the main findings. Studies were reviewed and highlighted using a hierarchy-of-evidence approach, where the best evidence is the focus of the synthesis for each Key Question. RCTs were prioritized and studies with lower risk of bias ratings were given more weight in our synthesis for each clinical indication and outcome. Since the Key Questions varied in nature and scope, the approach to synthesis also varied.

Quantitative data were summarized in summary tables and descriptive analysis and interpretation of the results are provided. Meta-analyses were not performed as they would not produce meaningful results due to limited numbers of studies reporting similar outcomes and due to heterogeneity based on study design, patient population, and interventions. Most trials lacked statistical power to detect differences in outcomes between intervention and comparison groups, limiting further comparisons and interpretation of results. We created categories of results based primarily on the direction of the effect and whether differences were statistically significant. Results are summarized across studies grouped by preventive service and/or telehealth function/modality.

Qualitative data were summarized in tables. For synthesis of qualitative data on barriers, facilitators, and patient preferences (KQ 1b, 1e, 2b, 2e), key statements addressing included outcomes were extracted from each study and categorized according to theme and type of preventive service (family planning, contraception, STI counseling, IPV), and results were summarized in tables.

There were not sufficient data available for any of the KQs to conduct an additional analysis of populations particularly affected by potential barriers to preventive services and telemedicine. In addition, outcomes related to health equity, access, and disparities were considered for inclusion but were not reported by studies.

Grading the Strength of the Body of Evidence

The strength of evidence (SOE) was assessed as high, moderate, low, or insufficient, using the approach described in the AHRQ Methods Guide,⁴⁸ based on study limitations, consistency, directness, precision, and reporting bias. These criteria were applied regardless of whether evidence was synthesized quantitatively or qualitatively. SOE was initially assessed by one researcher and confirmed by a second. Descriptions of criteria and overall grades are described in full in Appendix A.

SOE and the corresponding conclusions are expressed in terms of whether the outcome measured and analyzed in the studies is better, worse, or similar with telehealth compared with in-person clinical interactions without telehealth, often referred to in studies as usual care. However, usual care could have different definitions depending on the study, including in-person interactions; interactions providing enhanced versus routine counseling; generic information; information covering other health topics; or no clinical interaction. For this reason, we have provided detailed descriptions of usual care when they were included in the articles.

KQs 1b, 1d, 1e, 2b, 2d, and 2e are descriptive. When applicable, a formal SOE assessment was conducted based on study-design specific criteria. We prioritized reports of U.S. national or regional studies over local reports or data from other countries. We summarized the strengths and limitations of the data collection and analyses of the included reports for these questions, with a focus on elements such as the extent the sample represents the population of interest and the completeness and reliability of the data.

The evidence for KQs 1b, 1d, 1e, 2b, 2d, and 2e was limited and consisted of studies that used qualitative methods (e.g., interviews, case studies, focus groups) as well as quantitative methods and the studies were not comparative. We assessed SOE based on methodological limitations, coherence, adequacy, and relevance. We recognize that studies conducted or published quickly during the pandemic may contribute to overall conclusions, but may not be as rigorous as a study of the same design conducted during other timeframes. This was taken into consideration when considering the body of evidence.

Assessing Applicability

Applicability was considered according to the approach described in the Methods Guide for Effectiveness and Comparative Effectiveness Reviews.⁴⁸ We used the PICOS framework to consider the applicability of the evidence base for each Key Question, for example, examining the characteristics of the patient populations (e.g., clinical condition) and study setting to determine how well the identified body of evidence matches these criteria. Information relevant for assessing applicability included the number and diversity of settings or locations as well as characteristics of the population, telehealth intervention, or implementation strategy.⁵⁵ Variability in the studies may limit the ability to generalize the results to other populations or settings and affect the degree of confidence on how well this evidence base can be applied to other populations and settings.

Peer Review and Public Commentary

Experts were invited to provide external peer review of this systematic review; AHRQ and the Health Resources and Services Administration (HRSA) also provided comments. In addition, the draft report was posted for public comment on the AHRQ website for 4 weeks. All comments were reviewed and used to inform revisions for the final report.