Evidence Brief: Comparative Effectiveness of Appointment Recall Reminder Procedures for Follow-up Appointments

Kim Peterson; Ellen McCleery; Johanna Anderson; Kallie Waldrip; Mark Helfand

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Evidence Brief: Comparative Effectiveness of Appointment Recall Reminder Procedures for Follow-up Appointments

Kim Peterson, MS, Ellen McCleery, MPH, Johanna Anderson, MPH, Kallie Waldrip, MS, and Mark Helfand, MD, MPH, MS.

Washington (DC): Department of Veterans Affairs (US); 2015 Jul.

PREFACE

The VA Evidence-based Synthesis Program (ESP) was established in 2007 to provide timely and accurate syntheses of targeted healthcare topics of particular importance to clinicians, managers, and policymakers as they work to improve the health and healthcare of Veterans. QUERI provides funding for four ESP Centers, and each Center has an active University affiliation. Center Directors are recognized leaders in the field of evidence synthesis with close ties to the AHRQ Evidence-based Practice Centers. The ESP is governed by a Steering Committee comprised of participants from VHA Policy, Program, and Operations Offices, VISN leadership, field-based investigators, and others as designated appropriate by QUERI/HSR&D.

The ESP Centers generate evidence syntheses on important clinical practice topics. These reports help:

Develop clinical policies informed by evidence;
Implement effective services to improve patient outcomes and to support VA clinical practice guidelines and performance measures; and
Set the direction for future research to address gaps in clinical knowledge.

The ESP disseminates these reports throughout VA and in the published literature; some evidence syntheses have informed the clinical guidelines of large professional organizations.

The ESP Coordinating Center (ESP CC), located in Portland, Oregon, was created in 2009 to expand the capacity of QUERI/HSR&D and is charged with oversight of national ESP program operations, program development and evaluation, and dissemination efforts. The ESP CC establishes standard operating procedures for the production of evidence synthesis reports; facilitates a national topic nomination, prioritization, and selection process; manages the research portfolio of each Center; facilitates editorial review processes; ensures methodological consistency and quality of products; produces “rapid response evidence briefs” at the request of VHA senior leadership; collaborates with HSR&D Center for Information Dissemination and Education Resources (CIDER) to develop a national dissemination strategy for all ESP products; and interfaces with stakeholders to effectively engage the program.

Comments on this evidence report are welcome and can be sent to Nicole Floyd, ESP CC Program Manager, at vog.av@dyolF.elociN.

EXECUTIVE SUMMARY

After 5 years of mandated use, the 2010 Recall Reminder policy is being revisited because significant decreases have not been observed from 2010 to 2014 in overall appointment no-shows or cancellations and staff members have criticized the time and resource intensity of the system. We found published literature to be of little help in deciding whether to keep, modify, or replace the current Class I VHA recall reminder software. This is because the published literature has not evaluated the effect of alternatives on a complete set of related system outcomes: no-show rates, backlog, call center waiting time, administrative burden, and patient satisfaction. It also has not evaluated policy options such as:

More flexibility. This option could give clinicians and patients a choice of recall reminder or scheduling a future appointment >90 days, more options for mode and threshold of notification, and the option to contact a dedicated phone line in the pertinent clinic versus a general call center number with redirection to specific clinics.
Adaptation to local circumstances. This option may involve using a different cutoff than 90 days depending on a facility's backlog of new patients, call center hold time, or the patient population.

The studies we identified (Table A) provided limited evidence that use of a recall reminder scheduling system can decrease the rate of missed appointments by 7 percentage points compared to ‘365 scheduling’ in elderly Veterans. However, this evidence did not evaluate resource use or the impact on backlog, and the specific circumstances and practices of the medical center in which the study was performed are not generalizable to the VA health system overall. Studies that compared appointment reminder methods, regardless of scheduling method or whether patients were new or established, suggest that the Access and Clinic Administration Program (ACAP) should explore the use of live telephone reminders and text message reminders as an alternative to the current postal reminders. However, the independent effect of the scheduling component versus the reminder component remains unknown. Directions for future VA quality improvement initiatives include evaluation of (1) a complete set of pertinent and related system outcomes, (2) policy options of more flexibility and adaptation to local circumstances, (3) the impact of potential patient, provider, and system effect modifiers, (4) the impact of variation in recall reminder scheduling system design (ie, how and when Veterans are contacted), (5) the independent contributions from the scheduling and reminder components, (6) the use of agent-based models to identify areas with greatest potential for change, and (7) tailoring the scheduling approach to the individual Veteran.

Table A

Main Findings.

Background

As part of the VHA's focus on improving Veteran access, the ACAP requested that the ESP CC conduct a rapid evidence brief to evaluate the comparative effectiveness of appointment recall reminder procedures for follow-up appointments, by (1) considering their overall net benefit in reducing follow-up appointment missed opportunities, while not worsening organizational outcomes and (2) identifying potential effect modifiers. Findings from this evidence brief will be used to inform a potential revision of the 2010 VHA Directive 2010-027 that prohibits scheduling of >90-day follow-up appointments at the time Veterans leave the clinic and mandated the use of recall reminder procedures.

Methods

We searched MEDLINE®, the Cochrane Database of Systematic Reviews, and the Cochrane Central Registry of Controlled Trials using terms for appointments, reminders and schedules. We used prespecified criteria for rating internal validity and strength of the evidence for each outcome and comparison. See our PROSPERO protocol for our full methods.

INTRODUCTION

PURPOSE

As part of the VHA's focus on improving Veteran access, the ESP Coordinating Center (ESP CC) is responding to a request from the Acting Deputy Under Secretary for Health for Operations Management (DUSHOM) through the Access and Clinic Administration Program (ACAP) for an evidence brief on the comparative effectiveness of appointment recall reminder procedures for established patients returning for follow-up appointments. Recall appointments are defined as future patient appointments in which the patient needs to be seen in more than 90 days. The main purpose of this brief review is to summarize the evidence on the comparative effectiveness of the current Class I VHA recall reminder software, alternative recall reminder software or approaches, and scheduling follow-up appointments at the time of leaving the office in reducing missed opportunities for Veteran follow-up appointments without negatively impacting opportunity costs.

The ACAP will use the findings from this evidence brief to help inform refinement of clinical manager training development and scheduling policies, processes, and standard operating procedures (SOPs). In addition, findings will drive recall reminder software development intended to increase the ease of patient scheduling, decrease patient no-show rates, cancellation rates, and loss to follow-up, and enhance health care delivery and access.

BACKGROUND

Missed health care appointments are a major source of potentially avoidable cost and resource inefficiency that can adversely affect organizational workflow and increase clinic wait times. Missed appointments also may reflect needed care that was not delivered that can result in delays in diagnosis and appropriate treatment and decrease patient health outcomes.¹^-⁴

In 2008, an audit by the Office of Inspector General (OIG) found that the Veterans Health Administration's (VHA) efforts to reduce unused outpatient appointments were inadequate and recommended establishment of procedures to (1) measure and track unused outpatient appointments, (2) measure the effectiveness of processes for reducing missed opportunities and implement best practices nationwide, and (3) require facility directors to ensure unused appointments are used.⁵ A few examples of VHA efforts to address the 2008 OIG's recommendations include (1) 2010 implementation of a standardized computerized system for tracking and reducing missed opportunities for >90-day follow-up appointments (Class I Recall Reminder software) and (2) in 2011, the Pittsburgh Healthcare System Veterans Engineering Resource Center (VAPHS VERC) developed the National Initiative to Reduce Missed Opportunities (NIRMO) for tracking missed opportunities, understanding and analyzing factors that predict them, and developing and deploying strategies for improvement.

In the VA, the farther out an appointment is scheduled, the less likely the appointment will happen (45%-60% for appointment age >90 days vs 70-80% for appointment age <14 days; J. Prentice, S.D. Pizer, unpublished data, 2015). Starting in 2010, VHA Directive 2010-027 prohibited continued scheduling of greater than 90-day follow-up appointments at the time Veterans were leaving the clinic and mandated the use of strategies for contacting patients closer to the time of the needed visit to remind them to schedule the appointment (‘Recall Reminder’).⁶ Under this system, when a patient checks out after seeing a provider, a future appointment is made only if the patient is to return to clinic within 90 days. Otherwise, (1) VA staff use the software to schedule the Recall Reminder, (2) 2-4 weeks prior to the recall date, software automatically notifies clinic staff that it is time to remind the Veteran of the need to schedule a follow-up visit, (3) a mail or phone reminder is sent to the patient, and either (4) the patient calls to schedule, or (5) patient does not call to schedule and goes onto a delinquency list and has to be contacted by VA staff.

After 5 years of mandated use, the 2010 Recall Reminder policy is being revisited because VHA Support Service Center data have shown no significant decreases from 2010 to 2014 in no-shows overall (7.1% vs 7.4%), canceled by clinic rate (9.2% vs 9.9%), or canceled by patient rate (14.8% vs 15.0%) and some staff using the Recall Reminder system have criticized the system, asserting that it is very time- and resource-intensive (written communication, March 2015). For example, as a result of requiring Veterans to call in to schedule their follow-up appointment rather than scheduling the follow-up as they are leaving the office, the VISN 8 Call Center reported a 10% increase in their call volume from 2010 to 2014 (written communication, March 2015). Another common complaint from the field is that manual management of the delinquency list is very labor-intensive (VISN3, email communication, December 24, 2014). Other potential unintended consequences of using the Recall Reminder system include negative impacts on patient follow-up, Veteran satisfaction (if Veterans want to leave with an appointment), cost and other organizational outcomes (eg, productivity, turn over, grievances, training requirements, infrastructure requirements, etc). Recall Reminders could be implemented in several different ways, potentially leading to increased or decreased scheduler burden and other adverse events, so how best to implement the system is an important question. For example, if sending Veterans due for follow-up appointments a notice at 60 days, 30 days, and calling them at 20 days, 18 days, and 15 days doesn't significantly reduce no-shows beyond a less intensive approach of calling Veterans at 18 days and 15 days, then the less intensive approach may be preferable due to the decreased workload burden.

Although NIRMO has not yet evaluated the specific impact of the Recall Reminder system, they surveyed Veterans about reasons for missed appointments and VA staff regarding scheduling practices and implementation of strategies to reduce missed appointments.⁷ NIRMO's survey of 4,749 Veterans found that the top reported reasons for missed appointments were that they forgot (19%), miscellaneous (reasons other than those listed, 16%), not aware of the appointment (15%), no transportation (8%), poor weather (7%), sick (7%), something unexpected came up (7%), and cancelled the appointment beforehand (5%). NIRMO also surveyed 1,493 VA staff to identify scheduling practices and strategies used to reduce missed appointments. Compared to a missed appointments rate of 18.7% for the strategy of negotiating appointments only, reminder calls reduced the missed appointments rate to 16.0%. The rate was reduced to 14.7% with both reminder calls and promotion of provider continuity, and to 13.7% with reminder calls, promoting provider continuity, and receiving appointment cancellations through main facility phone line instead of individual clinic phone line.

Taking into account Veteran and staff surveys, NIRMO has recommended 10 general strategies for reducing missed appointments: (1) cancel appointments as they are received, (2) negotiate all appointments, (3) coordinate appointments with transportation and other appointments already scheduled, (4) offer open access or same-day access, (5) use nontraditional modes of care, (6) manage the schedule to ensure clinics run on time, (7) improve interactions between patients and providers, patients and clerks, and between staff members, (8) disseminate educational posters, (9) use the Recall Reminder software to schedule follow-up visits beyond 90 days, and (10) perform live targeted reminder calls to patients that they've identified as having a 20% or greater no-show probability using a predictive model they developed.

VHA is taking several steps to determine how to optimize the use of Recall Reminder processes. First, the VA is undertaking a quality improvement initiative that will evaluate the effectiveness of various Recall Reminder approaches across 6 pilot sites, comparing sequence and timing of reminder postcards and calls and the traditional approach allowing Veterans to schedule appointments before leaving the office (up to a year in advance). Second, through the ACAP, the DUSHOM requested that the ESP CC conduct a rapid evidence brief to evaluate the comparative effectiveness of the current Class I VHA recall reminder software, alternative recall reminder software or approaches, or scheduling follow-up appointments at the time of leaving the office in reducing missed opportunities for established Veteran follow-up appointments without negatively impacting opportunity costs.

SCOPE

The objective of this evidence brief is to synthesize the literature on the comparative effectiveness of appointment recall reminder systems. The ESP Coordinating Center investigators and representatives of the Access and Clinic Administration Program (ACAP) worked together to identify the population, comparator, outcome, timing, setting, and study design characteristics of interest. The ACAP approved the following key questions and eligibility criteria to guide this review:

KEY QUESTIONS

Key Question 1: For adult patients who are targeted for follow-up appointments, what is the comparative effectiveness of the current Class I VHA recall reminder software, alternative recall reminder software or approaches, or scheduling follow-up appointments at the time of leaving the office?

Key Question 2: For adult patients who are targeted for future appointments, does the comparative effectiveness of Appointment Recall Reminder (RR) procedures versus other kinds of follow-up appointment scheduling systems differ according to:

Patient factors: Preference, clinical characteristics
Appointment scheduling systems engineering design and management factors: Mode of notification (mail, phone, electronic), threshold for notification (1 month, 2 weeks), mode of patient response, reminder type
Facility characteristics: Efficiency, backlogs.

ELIGIBILITY CRITERIA

The ESP included studies that met the following criteria:

Population: Adult patients who are targeted for follow-up appointments.
Intervention: Any procedures for scheduling established patients' follow-up appointments. We accepted any type of procedures. These included, but were not limited to strategies incorporating the following procedures:
1. 365 scheduling: Negotiation of follow-up visit upon leaving the clinic, regardless of how far in the future the appointment is needed;
2. Strategies for reducing appointment age – after patients leave the office, contacting the patients closer to the time the future appointment is needed by:
  1. Recall Reminder: Sending a notification requesting that the patient contact the office to schedule an appointment
  2. Blind scheduling: Sending a notification of an appointment that has been scheduled on behalf of the patient without their input about preference on date or time.
  The highest-priority studies were those that most directly addressed our questions about the comparative effectiveness of different systems for scheduling established patients' follow-up appointments, with or without reminders. To address gaps in the highest-priority evidence, we also accepted lower-priority studies that either (a) focused on scheduling new patients for initial visits or (b) focused only on the reminder component.
Comparison: Mandated versus flexible use of a recall reminder system; comparison of different recall reminder engineering designs; comparison of recall reminder versus other strategies for reducing appointment age.
Outcomes: Primary outcomes of interest include no-show rates and cancel rates. Secondary outcomes of interest include appointment wait times, patient loss to follow-up (undelivered needed care), scheduler learning and behavior, organizational outcomes (eg, productivity, turnover, grievances, training, infrastructure requirements), and patient satisfaction.
Timing: No restrictions.
Setting: Within and outside of the VA. We will prioritize VA studies, but will look outside of the VA to fill gaps in VA evidence, including international studies.
Study design: Longitudinal studies. Using a Best Evidence approach, we will prioritize evidence from systematic reviews and multisite studies that adequately controlled for potential patient-, provider-, and system-level confounding factors. Inferior study designs (eg, single-site, inadequate control for confounding) will only be accepted to fill gaps in higher-level evidence.

We are aware of the large volume of evidence on the effectiveness of reminders for increasing vaccine and screening test uptake. However, we excluded those from this review because they generally focus on ultimate uptake regardless of whether it encompassed multiple failed appointment attempts and/or the procedures can sometimes be completed in walk-in clinics and don't always involve any scheduling.

ANALYTIC FRAMEWORK

The analytic framework below illustrates the populations, interventions, outcomes, and adverse effects that guided this review and their relationship to the Key Questions.

Figure 1

Analytic Framework.

METHODS

To identify relevant citations, our research librarian searched MEDLINE®, the Cochrane Database of Systematic Reviews, and the Cochrane Central Registry of Controlled Trials on March 5, 2015 and April 7, 2015 using terms for appointments, reminders and schedules. As our initial review of the search results found numerous systematic reviews that adequately covered the literature through 2010, we limited our search for primary literature to articles published in or after 2010. The exact search strategies are provided in the supplemental materials. To rate the internal validity of included studies, we used Cochrane's Risk of Bias Tool for controlled trials and the Drug Effectiveness Review Project's Tool for observational studies.⁸^,⁹ We graded the strength of the overall body of evidence using the AHRQ Methods Guide for Comparative Effectiveness Reviews which is based on the risk of bias of individual studies (study design and internal validity), consistency, directness, and precision. We followed the February 2015 AHRQ Guidance for determining when strength of evidence rating from existing systematic reviews could be used and for determining whether new primary studies would change the strength of the evidence or conclusions of previous reviews.¹⁰ However, for lower-priority existing reviews that either (a) focused on scheduling new patients for initial visits or (b) focused only on the reminder component (see eligibility criteria above), to fit the abbreviated timeline of this rapid review, we modified the AHRQ-recommended approach for rating strength of evidence. When lower-priority existing reviews did not complete strength of evidence ratings and did not provide adequate detail about the primary studies for us to do so, we described the strength of the evidence as unclear and noted which details were lacking.

The complete description of our full methods can be found on the PROSPERO international prospective register of systematic reviews website (http://www.crd.york.ac.uk/PROSPERO/; registration number CRD42015020654).

RESULTS

LITERATURE FLOW

We screened 529 unique records and included 25 articles in this evidence brief (Figure 2): 13 systematic reviews and 12 primary studies. For Key Question 1, we only identified 2 flawed single-site, non-concurrently controlled cohort studies that compared different approaches to scheduling follow-up appointments¹¹^,¹² and one systematic review that compared different methods of scheduling initial appointments.¹³ Of the 2 follow-up appointment scheduling cohorts, one was a VA study, but because it focused on a single geriatric clinic, its findings may have limited applicability to the broader VA population.¹¹

Figure 2

Literature flow chart. * ≠ 25, one article addresses more than one Key Question.

Because neither study of scheduling future appointments addressed the effects of differences in patient factors, engineering design, or facility characteristics (Key Question 2), we looked at studies of these factors on the effectiveness of reminders for existing appointments. Due to the large volume of systematic reviews on reminders for existing appointments available that evaluated the primary literature through 2010,¹^-³^,¹³^-²² we only included primary studies on reminders for existing appointments published from 2010 onward. Table 1 shows which reminder types are compared in the included systematic reviews and primary studies.

Table 1

Reminder types included in systematic reviews and primary studies.

We identified 13 systematic reviews and 12 primary studies that met our inclusion criteria. Four of the primary studies are randomized controlled trials,²³^-²⁶ 7 are non-concurrent cohort studies,¹¹^,¹²^,²⁷^-³¹ and one is an uncontrolled before-after study.³²

We rated most of the included systematic reviews as fair or good quality.²^,³^,¹³^-¹⁵^,¹⁷^-¹⁹^,²¹ We rated 3 systematic reviews as poor quality for providing insufficient detail and not rating the quality of included primary studies.¹⁶^,²⁰^,²² We rated 3 of the 4 included RCTs as low risk of bias²³^,²⁵^,²⁶ and one as medium risk of bias²⁴ since allocation concealment was not described. We rated 7 of the 8 observational studies as poor quality for not accounting for temporal trends¹¹^,¹²^,²⁷^-²⁹^,³¹^,³² and one as fair quality.³⁰ Three included primary studies were conducted at VA medical centers: one at the Miami VA Geriatrics Clinic,¹¹ one at 3 HIV primary care clinics in the Los Angeles VA system,³⁰ and one at the Providence VA Medical Center homeless primary care clinic.³²

KEY QUESTION 1. For adult patients who are targeted for follow-up appointments, what is the comparative effectiveness of appointment Recall Reminder (RR) procedures versus other kinds of follow-up appointment scheduling systems?

Comparison of different methods for follow-up patient scheduling

We only identified 2 non-concurrently controlled studies that compared different methods scheduling established patients for follow-up visits.¹¹^,¹² The more relevant of the 2 studies¹¹ demonstrated that using a recall reminder system to schedule follow-up appointments in the Miami VA Geriatric Clinic in fiscal year 2006 decreased the rate of missed appointments from 18% to 11% (P=.000) compared to ‘365 scheduling.’ But the usefulness of this benefit is unclear in the absence of accompanying information about how the recall reminder impacted other key factors of patient follow-up, scheduler time burden, or organizational outcomes. Also, because this study lacked important system design details (eg, methods for managing recall delinquencies, letter content, etc), the applicability of its findings to the current Class I VHA recall reminder software is unclear.

In fiscal year 2006, the Miami VA Geriatrics Clinic demonstrated a decrease in rate of missed appointments from 18% to 11% (P=.000) and rate of patients with wait times longer than 1 month (15% vs 0%) after changing their system of scheduling future appointments from the traditional model of scheduling at the end of each visit to sending patients a letter advising them to call and make an appointment 30 days before their next anticipated visit.¹¹ The reduction in missed appointments appeared to be independent of any underlying trends over time. The monthly missed appointment rate was fluctuating between 14% and 25% in the year prior to implementation with no clear pattern, compared to a range of 6% to 18% in the year after implementation. However, the lack of a concurrent control group still prevents us from ruling out the potential confounding effects of other organizational changes, such as staffing changes. The between-group difference may also have been confounded by differences in unmeasured patient and appointment characteristics. Although this was a study of a Veteran population, its applicability is still limited because it only involved a single site of geriatric patients. Finally, we don't know exactly to what type of geriatric patients, appointments, or reminder letters these findings apply since those details were not provided.

The second study compared non-concurrent cohorts of 2,116 follow-up appointment patients from an ophthalmology practice at Dartmouth-Hitchcock Medical Center who were either sent a reminder postcard advising patients to call and make a follow-up appointment or sent a computer-generated letter of a “blind” scheduled appointment 4 weeks before the appointment. Compared to the blind scheduled group, the reminder postcard group had a slightly lower rate of no-shows (absolute difference of -2%; 4.5% compared with 6.5%; P=.09).¹² However, the blind-scheduling approach had the advantages of keeping more patients in contact with the office (100% vs 56%) and increasing estimated billing revenue for the first year by $74,878. Patient satisfaction and scheduler effort were similar between the 2 approaches (data not reported).¹² However, because blind scheduling is not an alternative the VA is considering and this study is in a narrow non-VA population, the applicability of these findings to the broad VA population is likely low.

Other methodological limitations include unreliable methods for analyzing no-shows and potential confounding. For the analysis of no-shows, although the publication reported a significantly larger increase in no-shows for the blind scheduled group (+4.5%; P<.0001), we believe this was an overestimation. The no-show rate for the reminder postcard group cited in the paper (2%) appears to have been calculated based on a denominator that included 463 patients who never made a reappointment in the first place. Therefore, we used a no-show rate of 4.5% for the reminder postcard that was based only on patients who made an appointment (N=599). However, regardless of how the no-shows were calculated, another possible explanation for the higher rate in the blind scheduled group may be the difference in season of scheduling. Compared to the postcard group who were contacted in spring, the blind scheduled group was scheduled in the summer months, where appointment attendance may be less reliable in general due to higher rates of vacation and recreation. Also, for the postcard group, there was no information on the amount of time between when they called to make the appointment and the scheduled date of the appointment (ie, appointment age). If the appointment age was lower for the postcard group compared to the 4 weeks for the blind scheduled group, this may have contributed to the higher no-show rate in the blind scheduled group. Finally, the between-group difference may have been confounded by differences in unmeasured patient and appointment characteristics. Reasons for reappointments ranged from contact lens checks to ocular surgery follow-up, but there was no analysis of whether no-shows varied based on appointment type. Also, the lack of detail about the content of the postcard reminder limits it applicability. We know that the cards were inscribed by hand and included the phone number, but we do not know the nature of the message.

Comparison of different methods for new patient scheduling

Because reasons for no-shows may be different between new and established patients, we considered evidence on the comparative effectiveness of different scheduling practices for new patients to be indirect to answering our questions about impact on established patients. However, because we found very limited evidence that compared different methods of scheduling established patients for follow-up appointments, we also explored the evidence on scheduling new patients for any useful insights. A good-quality systematic review provided insufficient evidence of the overall comparative effectiveness for different scheduling approaches.¹³ Although there is low-strength evidence of similar attendance rates for blind versus patient-initiated scheduling and accelerated versus standard access, risk of potential adverse effects was not reported.¹³ The authors of this systematic review searched through June 2012 and we did not find any more recent primary studies.

Blind scheduling versus patient-initiated

Three small single-site UK RCTs (N=451) provided low-strength evidence that requiring patients to contact the clinic to make an appointment did not consistently improve attendance at the initial mental health appointments compared with a blind scheduling approach.¹³ Effects on attendance ranged from a slight decrease from 48% to 45% (RR 0.94; 95% CI 0.70 to 1.26) in a marital and sexual difficulties clinic to an increase from 67% to 79% (RR 1.67; 95% CI, 1.06 to 2.55) in a specialist psychotherapy clinic. The risk of bias for these studies was described as “mainly unclear.”

Accelerated access

Two small single-site RCTs from substance abuse clinics in the US (N=245) provided low-strength evidence that offering appointments on the same day or within 48 hours did not improve initial appointment attendance compared with scheduling appointments further in the future.¹³ In a community-based substance abuse agency, attendance increased from 41% to 65% (RR 1.60; 95% CI, 0.91-2.82) when participants were offered same-day appointments and discussed their obstacles to attending appointments compared to standard scheduling with unspecified average wait times. In a substance abuse research clinic, attendance increased from 57% to 67% (RR 1.18; 95% CI, 0.79 to 1.76) when appointments were scheduled within 48 hours compared with within an average of 5 days.

KEY QUESTION 2. For adult patients who are targeted for future appointments, does the comparative effectiveness of appointment Recall Reminder (RR) procedures versus other kinds of follow-up appointment scheduling systems differ according to patient factors, scheduling system engineering design and management factors, or facility characteristics?

We did not identify any evidence on how the comparative effectiveness of appointment recall reminder procedures versus other kinds of follow-up appointment scheduling systems differ according to other factors. But below we summarize evidence on how the comparative effectiveness of reminders for existing appointments differs according to other factors. In general, there is low-strength evidence that reminders may be less effective in new patients compared with established patients and in patients with depression, but no evidence that the comparative effectiveness of reminders for existing appointments differs by other individual patient factors (such as gender, deprivation status, employment status, substance abuse, ethnicity, mental health, other health problems, symptomatic compared with non-symptomatic health status, diagnostic stage, or perceived severity of the patient's health condition).

A. Patient factors

A1. New versus established patients

One RCT provides low-strength evidence that new patients had higher no-show rates compared with established patients after a telephone reminder from clinic staff or after an automated telephone reminder in an outpatient multispecialty practice, but that there was no difference in no-show rates between new and established patients among the group that did not receive a reminder.²⁵

A2. Patient clinical factors

One non-concurrent cohort study in a VA HIV primary care clinic provides low-strength evidence that a diagnosis of depression may moderate the effectiveness of an appointment reminder. Adding an automated telephone reminder 2 weeks before the appointment to standard appointment reminders 3 days before the appointment was effective in reducing no-shows among patients without depression (18.2% vs 23.4, p<.05), but not effective in reducing no-shows among patients with depression (30.9% vs 24.9, p>.05).³⁰

A3. Patient preference

We did not identify any studies that reported on how the comparative effectiveness of existing appointments differs by patient preference. The association between the mode of the appointment reminder and participant satisfaction is discussed below.

A4. Patient age

One systematic review concluded that the use of SMS reminders was related to an increased appointment attendance compared with no reminder and that there were no significant subgroup differences by target age group (pediatric versus older).¹⁷ However, it is unclear how generalizable these results are to the VA, since the age group “older than pediatric” was not broken down further to examine how the effectiveness of SMS reminders varies.

A5. Other patient factors

One systematic review reported insufficient evidence to draw conclusions about whether the comparative effectiveness of reminders for existing appointments differs by other individual patient factors (such as gender, deprivation status, employment status, substance abuse, ethnicity, mental health, other health problems, symptomatic compared with non-symptomatic health status, diagnostic stage, and perceived severity of the patient's health condition).³ The systematic review cites only one study in an orthodontic practice that evaluated how gender and age affects attendance and concludes that this evidence is insufficient to make any conclusions about how patient characteristics mediate the association between a reminder and attendance outcomes.

B. Appointment scheduling systems engineering design and management factors

B1. Mode of notification

We found no studies that compared different modes of notification within the same recall reminder system.

B2. Threshold for notification (1 month, 2 weeks)

Previous systematic reviews provide consistent evidence that appointment attendance does not clearly vary based on differences in timing of the SMS text reminders (24, 24, and 72+ hours),¹⁷ postal reminders (1 vs 3 days),²¹ or telephone reminders (1-7 days).¹⁸ However, the strength of the evidence from the previous reviews is unclear as they provided insufficient detail to evaluate how variation in primary study quality may have affected outcomes or the precision of the estimates.

B3. Mode of patient response

We found no studies that compared different modes of patient response within the same recall reminder system.

B4. Reminder mode

Regardless of scheduling method, the evidence does not demonstrate that any particular reminder type (ie, phone, postal, or text) has a clear net benefit over any other (Table 2). One large, good-quality RCT (N=8,071) provides moderate-strength evidence that a live telephone reminder from clinic staff 2 days prior to an outpatient specialty appointment significantly reduced no-shows compared to an automated telephone reminder. But there was no difference in cancellation rate between the 2 groups and impact on our other secondary outcomes of interest was not evaluated.²⁵ Text messaging may be a potential alternative to live telephone reminder calls. A good-quality systematic review of 3 fair-quality RCTs of from China and Malaysia (N=2,509) consistently found that text message reminders result in similar attendance rates and cost less than telephone reminders.² However, a larger and more recent RCT from a Swiss urban academic primary care clinic (N=5,200) found higher no-shows in the group sent an automatic text-message reminder via ‘Easy SmartCare’ software 24 hours before their appointment compared to live telephone reminder calls.²⁴ Because we have very little information about the patients, appointment types, and reminder content in any of the studies, we cannot determine the reason for the differing results in the Asian and Swiss studies.

Table 2

Comparative effectiveness of reminder modes for existing appointments (Strength of Evidence: ●●●=High, ●●○=Moderate, ●○○=Low).

Postal reminders have been less widely studied than phone and text reminders. Low-strength evidence from 3 small RCTs (N=326) included in one systematic review suggests that postal reminders do not improve attendance to outpatient mental health appointments for people with serious mental illness.²¹ Another review concluded that postal reminders reduced the nonattendance rate to 7.6% on average in 6 RCTs and 1 historically-controlled cohort (range -3% to -17%; N= 6,621) in a variety of settings (orthodontics, women's health, general outpatient, colposcopy).²² Still, it is unclear whether this finding is reliable since the review did not account for potential differences in quality across the included studies and we could not judge the precision of the average attendance rates because measures of variance were not reported.

We rated the remainder of the evidence that compared different single-mode reminders as insufficient or unclear due to each comparison being supported by either one small study with medium to high limitations, or by systematic reviews that provided insufficient detail on included primary studies to rate their strength.

Few studies have evaluated how reminders using a combination of modes compare to single modes or no reminders. There is low-strength evidence that adding a text message to a postal reminder improved attendance at ear, nose, and throat clinics in one UK district general hospital (94% vs 86%; RR=1.10, 95% CI, 1.02-1.19) compared to a postal reminder alone.² The systematic review that included this study did not report on any other outcomes. Evidence from one small RCT of 66 participants with unknown randomization and blinding methods provided insufficient evidence to draw conclusions about the combination of a text and telephone reminder compared with no reminder for outpatient mental health visits.²¹

C. Facility characteristics

One systematic review concluded that the effectiveness of text message reminders in increasing appointment attendance did not differ based on variation in clinic type (primary care vs hospital outpatient clinics).¹⁷ But the strength of the evidence is unclear as the review provided insufficient detail to evaluate how variation in primary study quality may have affected outcomes or the precision of the estimates.

SUMMARY AND DISCUSSION

We only identified 2 studies that compared different methods of scheduling established patients for follow-up visits. The more relevant of the 2 studies provided limited evidence that use of a recall reminder scheduling system can decrease the rate of missed appointments by 7 percentage points compared to ‘365 scheduling’ in elderly Veterans seen in the Miami VA Geriatric Clinic in fiscal year 2006. However, the study provided no information about the time and resource impact of the recall reminder system or its engineering characteristics (eg, methods for managing recall delinquencies, letter content, etc), or the specific circumstances and practices of the medical center in which the study was performed.

Studies that compared appointment reminder methods suggest that ACAP should explore the use of live telephone reminders and text message reminders as an alternative to the current postal reminders. Live telephone reminders decreased no-shows by 4 percentage points compared with automatic phone reminders in non-VA multispecialty clinics. Compared to live telephone reminders, text messaging resulted in similar to borderline higher no-shows and similar satisfaction, but had the benefit of decreasing costs. However, these studies provided no information about the scheduling method used or whether patients were new or established. The independent effect of the Recall Reminder scheduling versus the appointment reminder components remain unknown, because studies only evaluated the comparative effectiveness of either the scheduling component or the reminder component, without accounting for their potential interaction.

LIMITATIONS

The main methodological limitation of this evidence brief is that because of the shortened timeframe our strength of evidence ratings were limited for lower-priority evidence that (a) focused on scheduling new patients for initial visits rather than scheduling follow-up visits for established appointments or (b) focused only on the appointment reminder component, regardless of how the appointment got scheduled (see eligibility criteria). However, because the findings from the lower-priority evidence are less relevant, we do not think this limitation would affect our overall conclusions.

There are 3 main gaps in the published literature. First, it has not evaluated the effect of alternatives on a complete set of related system outcomes: no-show rates, backlog, call center waiting time, administrative burden, and patient satisfaction. Second, published literature has not evaluated policy options such as:

More flexibility. This option could give clinicians and patients a choice of recall reminder or scheduling a future appointment >90 days, more options for mode and threshold of notification, and the option to contact a dedicated phone line in the pertinent clinic versus a general call center number with redirection to specific clinics.
Adaptation to local circumstances. This option may involve using a different cutoff than 90 days depending on a facility's backlog of new patients, call center hold time, or the patient population.

Third, we were not able to distinguish the effect of the scheduling component versus the reminder component of the Recall Reminder system, because studies only evaluated either the scheduling component or the reminder component, without accounting for their potential interaction.

FUTURE RESEARCH

The Recall Reminder procedure is part of a complex system that affects or is affected by the clinic backlog, the call center, and other parts of clinic administration. Previous research has focused narrowly on the no-show rate and a few other outcomes, missing, for example, the effect of burdening the call system or of patients' satisfaction with scheduling.

A VA quality improvement initiative is currently underway evaluating the effectiveness of various Recall Reminder approaches across 6 pilot sites and comparing sequence and timing of reminder postcards and calls and the ‘365 scheduling approach’ of allowing Veterans to schedule appointments up to a year in advance before leaving the office.

A systems approach to designing research would provide a clearer picture of the relationship between inputs, changes, and effects (outputs) of the system and could make it easier to measure or predict the effect of introducing incremental changes or adaptations to local circumstances (Table 3). In conjunction with a data collection system plan, a subset of VA centers could be encouraged to identify variations that might work better for their centers, and flexible, rapid experimental research on the effectiveness of these changes given different inputs could be developed. Eventually, the VA could develop a comprehensive and validated agent-based model to simulate, and reduce the need for, experiments by helping to predict the possible impact of changes and tweaks to the RR system.

Table 3

Potential systems approach research design for evaluating the Recall Reminder system.

To inform this approach, we recommend also systematically collecting data directly from patients regarding the reliability of their transportation, scheduling preferences, satisfaction with appointment characteristics (eg, date, time, and provider), perceptions of purpose and need of appointment, relationship with provider, and other factors that have been found to predict missed opportunities. To address unanswered questions about other potential effect modifiers, we also recommend evaluating the impact of variation in system and provider factors, particularly clinic backlog level.

As VA data has shown that forgetfulness is the top reason for missed appointments, more study is needed to isolate the added value of different types of appointment reminders when combined with using a recall reminder system to schedule follow-up appointments closer to the date they are needed. More study is also needed on the impact of variation in recall reminder scheduling system design, such as whether there are differences between using postal, phone, or text message reminders to contact Veterans or whether there are differences in contacting Veterans one day, one week, one month, or longer before the needed appointment. A factorial designed study could isolate both how much of the benefit of a recall reminder system versus 365 scheduling may be due to scheduling the appointment just in time versus the type of appointment reminder used and variation in the recall reminder system design. For example, a potential design could involve comparison of 3 types of scheduling approaches (ie, 365 scheduling, recall reminder scheduling by postcard, recall reminder scheduling by phone) and 2 types of reminders (ie, letter, phone), for a total of 6 treatment combinations: (1) 365 scheduling plus postal reminder, (2) 365 scheduling plus phone reminder, (3) recall reminder scheduling by postcard plus letter reminder, (4) recall reminder scheduling by postcard plus phone reminder, (5) recall reminder scheduling by phone plus letter reminder, and (6) recall reminder scheduling by phone plus phone reminder.

Additionally, the VA may consider exploring more targeted use of established patients' follow-up appointment scheduling procedures depending on the no-show probability of individual patients as determined based on a validated prediction model. NIRMO and other VA studies have primarily suggested and evaluated use of no-show modeling to inform targeting of reminder calls and improvement of clinic efficiency through selective overbooking strategies.³³^,³⁴ However, the VA could consider piloting a study where a patient's method of follow-up appointment scheduling could be tailored based on no-show probability. For example, the Veterans with the lowest no-show probability could be offered the least labor-intensive option of 365 scheduling and Veterans with the highest-risk of no-show could be mandated for more intense scheduling approaches.

CONCLUSIONS

We found published literature to be of very little help in deciding whether to keep, modify, or replace the current Class I VHA recall reminder software. Published studies provided limited evidence that use of a recall reminder scheduling system can decrease the rate of missed appointments by 7 percentage points compared to ‘365 scheduling’ in elderly Veterans, but without also evaluating resource use or the impact on backlog. Once appointments are scheduled, evidence also suggests that ACAP consider exploring use of live telephone reminders and text message reminders as an alternative to the current postal reminders to reduce Veterans' forgetfulness of their appointments. However, the independent effect of the scheduling component versus the reminder component remains unknown. Directions for future VA quality improvement initiatives include evaluation of (1) a complete set of pertinent and related system outcomes, (2) policy options of more flexibility and adaptation to local circumstances, (3) the impact of potential patient, provider and system effect modifiers, (4) the impact of variation in recall reminder scheduling system design (ie, how and when Veterans are contacted), (5) the independent contributions from the scheduling and reminder components, respectively, (6) the use of agent-based models to identify areas with greatest potential for change, and (7) tailoring the scheduling approach to the individual Veteran.

REFERENCES

1.: Car J, Gurol-Urganci I, de Jongh T, Vodopivec-Jamsek V, Atun R. Mobile phone messaging reminders for attendance at healthcare appointments. Cochrane Database of Systematic Reviews. 2012;7 CD007458. [PubMed: 22786507]
2.: GurolUrganci I, de Jongh T, VodopivecJamsek V, Atun R, Car J. Mobile phone messaging reminders for attendance at healthcare appointments. Cochrane Database of Systematic Reviews. 2013;(12) [PMC free article: PMC6485985] [PubMed: 24310741]
3.: McLean S, Gee M, Booth A, et al. Targeting the Use of Reminders and Notifications for Uptake by Populations (TURNUP): a systematic review and evidence synthesis. Southampton UK: National Institute for Health Research; 2014. [PubMed: 25642537]
4.: McMullen MJ, Netland PA. Lead time for appointment and the no-show rate in an ophthalmology clinic. Clin Ophthalmol. 2015;9:513–516. [PMC free article: PMC4370946] [PubMed: 25834388]
5.: Audit of Veterans Health Administration's Efforts to Reduce Unused Outpatient Appointments. Washington, DC: Department of Veterans Affairs Office of Inspector General; 2008.
6.: VHA Directive 2010-027. VHA Outpatient Scheduling Processes and Procedures. Washington, DC: 2010.
7.: Goffman R. The Veterans Health Administration's Position Regarding Reducing Missed Opportunities; Paper presented at: Healthcare Systems Process Improvement; February 19, 2015; Orlando, FL. 2015.
8.: Higgins JP, Altman DG, Gotzsche PC, et al. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928. [PMC free article: PMC3196245] [PubMed: 22008217]
9.: McDonagh MS, Jonas DE, Gartlehner G, et al. Methods for the drug effectiveness review project. BMC Medical Research Methodology. 2012;12:140. [PMC free article: PMC3532217] [PubMed: 22970848]
10.: Robinson KA, Chou R, Berkman ND, et al. Integrating Bodies of Evidence: Existing Systematic Reviews and Primary Studies Methods Guide for Effectiveness and Comparative Effectiveness Reviews. Rockville, MD: 2008. [PubMed: 25834891]
11.: Cherniack EP, Sandals L, Gillespie D, Maymi E, Aguilar E. The use of open-access scheduling for the elderly. Journal for Healthcare Quality. 2007;29(6):45–48. [PubMed: 18232607]
12.: Saine PJ, Baker SM. What is the best way to schedule patient follow-up appointments? Joint Commission Journal on Quality & Safety. 2003;29(6):309–315. [PubMed: 14564749]
13.: Schauman O, Aschan LE, Arias N, Beards S, Clement S. Interventions to increase initial appointment attendance in mental health services: a systematic review. Psychiatric Services. 2013;64(12):1249–1258. [PubMed: 24036532]
14.: Atherton H, Sawmynaden P, Meyer B, Car J. Email for the coordination of healthcare appointments and attendance reminders. Cochrane Database of Systematic Reviews. 2012;8 CD007981. [PubMed: 22895971]
15.: Free C, Phillips G, Watson L, et al. The effectiveness of mobile-health technologies to improve health care service delivery processes: a systematic review and meta-analysis. PLoS Medicine / Public Library of Science. 2013;10(1):e1001363. [PMC free article: PMC3566926] [PubMed: 23458994]
16.: George A, Rubin G. Non-attendance in general practice: a systematic review and its implications for access to primary health care. Family Practice. 2003;20(2):178–184. [PubMed: 12651793]
17.: Guy R, Hocking J, Wand H, Stott S, Ali H, Kaldor J. How effective are short message service reminders at increasing clinic attendance? A meta-analysis and systematic review. Health Services Research. 2012;47(2):614–632. [PMC free article: PMC3419880] [PubMed: 22091980]
18.: Hasvold PE, Wootton R. Use of telephone and SMS reminders to improve attendance at hospital appointments: a systematic review. Journal of Telemedicine & Telecare. 2011;17(7):358–364. [PMC free article: PMC3188816] [PubMed: 21933898]
19.: Liu Q, Abba K, Alejandria MM, Sinclair D, Balanag VM, Lansang MA. Reminder systems to improve patient adherence to tuberculosis clinic appointments for diagnosis and treatment. Cochrane Database of Systematic Reviews. 2014;11 CD006594. [PMC free article: PMC4448217] [PubMed: 25403701]
20.: Macharia WM, Leon G, Rowe BH, Stephenson BJ, Haynes RB. An overview of interventions to improve compliance with appointment keeping for medical services. JAMA. 1992;267(13):1813–1817. [PubMed: 1532036]
21.: Reda S, Rowett M, Makhoul S. Prompts to encourage appointment attendance for people with serious mental illness. Cochrane Database of Systematic Reviews. 2012;8
22.: Stubbs ND, Geraci SA, Stephenson PL, Jones DB, Sanders S. Methods to reduce outpatient non-attendance. American Journal of the Medical Sciences. 2012;344(3):211–219. [PubMed: 22475731]
23.: Junod Perron N, Dao MD, Kossovsky MP, et al. Reduction of missed appointments at an urban primary care clinic: a randomised controlled study. BMC Family Practice. 2010;11:79. [PMC free article: PMC2984453] [PubMed: 20973950]
24.: Junod Perron N, Dao MD, Righini NC, et al. Text-messaging versus telephone reminders to reduce missed appointments in an academic primary care clinic: a randomized controlled trial. BMC Health Services Research. 2013;13:125. [PMC free article: PMC3623700] [PubMed: 23557331]
25.: Parikh A, Gupta K, Wilson AC, Fields K, Cosgrove NM, Kostis JB. The effectiveness of outpatient appointment reminder systems in reducing no-show rates. American Journal of Medicine. 2010;123(6):542–548. [PubMed: 20569761]
26.: Taylor NF, Bottrell J, Lawler K, Benjamin D. Mobile telephone short message service reminders can reduce nonattendance in physical therapy outpatient clinics: a randomized controlled trial. Archives of Physical Medicine & Rehabilitation. 2012;93(1):21–26. [PubMed: 22000821]
27.: Brannan SO, Dewar C, Taggerty L, Clark S. The effect of short messaging service text on non-attendance in a general ophthalmology clinic. Scottish Medical Journal. 2011;56(3):148–150. [PubMed: 21873719]
28.: Farmer T, Brook G, McSorley J, Murphy S, Mohamed A. Using short message service text reminders to reduce ‘did not attend’ rates in sexual health and HIV appointment clinics. International Journal of STD & AIDS. 2014;25(4):289–293. [PubMed: 23999939]
29.: Haufler K, Harrington M. Using nurse-to-patient telephone calls to reduce day-of-surgery cancellations. AORN Journal. 2011;94(1):19–26. [PubMed: 21722768]
30.: Henry SR, Goetz MB, Asch SM. The effect of automated telephone appointment reminders on HIV primary care no-shows by veterans. Journal of the Association of Nurses in AIDS Care. 2012;23(5):409–418. [PubMed: 22424961]
31.: Perry JG. A preliminary investigation into the effect of the use of the Short Message Service (SMS) on patient attendance at an NHS Dental Access Centre in Scotland. Primary Dental Care. 2011;18(4):145–149. [PubMed: 21968040]
32.: McInnes DK, Petrakis BA, Gifford AL, et al. Retaining homeless veterans in outpatient care: a pilot study of mobile phone text message appointment reminders. American Journal of Public Health. 2014;104 Suppl 4:S588–594. [PMC free article: PMC4151902] [PubMed: 25100425]
33.: Alaeddini A, Yang K, Reddy C, Yu S. A probabilistic model for predicting the probability of no-show in hospital appointments. Health Care Manag Sci. 2011;14(2):146–157. [PubMed: 21286819]
34.: Daggy J, Lawley M, Willis D, et al. Using no-show modeling to improve clinic performance. Health Informatics J. 2010;16(4):246–259. [PubMed: 21216805]

SUPPLEMENTAL MATERIALS

SEARCH STRATEGIES

SYSTEMATIC REVIEWS

Database: Ovid MEDLINE (April 7, 2015)

Reminder Systems/
“Appointments and Schedules”/
((recall adj3 remind$) or (remind$ adj3 system$)).mp.
(appointment$ adj3 remind$).mp
or/1-4
meta-analysis.pt.
meta-analysis/ or systematic review/ or meta-analysis as topic/ or “meta analysis (topic)”/ or “systematic review (topic)”/ or exp technology assessment, biomedical/
((systematic* adj3 (review* or overview*)) or (methodologic* adj3 (review* or overview*))).ti,ab.
((quantitative adj3 (review* or overview* or synthes*)) or (research adj3 (integrati* or overview*))).ti,ab.
((integrative adj3 (review* or overview*)) or (collaborative adj3 (review* or overview*)) or (pool* adj3 analy*)).ti,ab.
(data synthes* or data extraction* or data abstraction*).ti,ab.
(handsearch* or hand search*).ti,ab.
(mantel haenszel or peto or der simonian or dersimonian or fixed effect* or latin square*).ti,ab.
(met analy* or metanaly* or technology assessment* or HTA or HTAs or technology overview* or technology appraisal*).ti,ab.
(meta regression* or metaregression*).ti,ab.
(meta-analy* or metaanaly* or systematic review* or biomedical technology assessment* or bio-medical technology assessment*).mp,hw.
(medline or cochrane or pubmed or medlars or embase or cinahl).ti,ab,hw.
(cochrane or (health adj2 technology assessment) or evidence report).jw.
(comparative adj3 (efficacy or effectiveness)).ti,ab.
(outcomes research or relative effectiveness).ti,ab.
((indirect or indirect treatment or mixed-treatment) adj comparison*).ti,ab.
or/6-21
5 and 22
limit 23 to yr=“2010 - 2015”

Database: Cochrane Database of Systematic Reviews (March 5, 2015)

reminder or appointment$.ti,ab.
((recall or appointment) adj2 reminder$).ti,ab.
1 or 2

PRIMARY STUDIES

Database: Ovid MEDLINE and Cochrane Central Registry of Controlled Trials (March 5, 2015)

Reminder Systems/
1 not (child$ or pediatric$ or adolescen$).mp.
limit 2 to (clinical trial or comparative study or controlled clinical trial or journal article or randomized controlled trial)
2 and (random$ or control$ or cohort or compar$).mp.
3 or 4
“Appointments and Schedules”/
appointment$.ti,ab.
5 and (6 or 7)

LIST OF EXCLUDED STUDIES

PRIMARY STUDIES ON REMINDERS FOR EXISTING APPOINTMENTS PUBLISHED BEFORE 2010

Ahluwalia JS, McNagny SE, Kanuru NK. A randomized trial to improve follow-up care in severe uncontrolled hypertensives at an inner-city walk-in clinic. Journal of Health Care for the Poor & Underserved. 1996;7(4):377–389. [PubMed: 8908893]
Anderson RM, Musch DC, Nwankwo RB, et al. Personalized follow-up increases return rate at urban eye disease screening clinics for African Americans with diabetes: results of a randomized trial. Ethnicity & Disease. 2003;13(1):40–46. [PubMed: 12723011]
Barr JK, Franks AL, Lee NC, Antonucci DM, Rifkind S, Schachter M. A randomized intervention to improve ongoing participation in mammography. American Journal of Managed Care. 2001;7(9):887–894. [PubMed: 11570022]
Chaudhry R, Scheitel SM, McMurtry EK, et al. Web-based proactive system to improve breast cancer screening: a randomized controlled trial. Archives of Internal Medicine. 2007;167(6):606–611. [PubMed: 17389293]
Crane LA, Leakey TA, Ehrsam G, Rimer BK, Warnecke RB. Effectiveness and cost-effectiveness of multiple outcalls to promote mammography among low-income women. Cancer Epidemiology, Biomarkers & Prevention. 2000;9(9):923–931. [PubMed: 11008910]
Hull S, Hagdrup N, Hart B, Griffiths C, Hennessy E. Boosting uptake of influenza immunisation: a randomised controlled trial of telephone appointing in general practice. British Journal of General Practice. 2002;52(482):712–716. [PMC free article: PMC1314410] [PubMed: 12236273]
Kiefe CI, Heudebert G, Box JB, Farmer RM, Michael M, Clancy CM. Compliance with post-hospitalization follow-up visits: rationing by inconvenience? Ethnicity & Disease. 1999;9(3):387–395. [PubMed: 10600061]
Margolis KL, Nichol KL, Wuorenma J, Von STL. Exporting a successful influenza vaccination program from a teaching hospital to a community outpatient setting. AM GERIATR SOC. 1992;40(10):1021–1023. JCR: Journal of Clinical Rheumatology. [PubMed: 1401675]
Mayer JA, Lewis EC, Slymen DJ, et al. Patient reminder letters to promote annual mammograms: a randomized controlled trial. Preventive Medicine. 2000;31(4):315–322. [PubMed: 11006056]
Miller PL, McConnell C. Reducing appointment no-shows and same-day cancellations. NAHAM Management Journal. 1997;24(1):9–11. [PubMed: 10168975]
Miller SM, Siejak KK, Schroeder CM, Lerman C, Hernandez E, Helm CW. Enhancing adherence following abnormal Pap smears among low-income minority women: a preventive telephone counseling strategy. Journal of the National Cancer Institute. 1997;89(10):703–708. [PubMed: 9168185]
Mohler PJ. Enhancing compliance with screening mammography recommendations: a clinical trial in a primary care office. Family Medicine. 1995;27(2):117–121. [PubMed: 7737444]
Moran WP, Nelson K, Wofford JL, Velez R. Computer-generated mailed reminders for influenza immunization: a clinical trial. Journal of General Internal Medicine. 1992;7(5):535–537. [PubMed: 1403212]
Norman P, Conner MT, Willits DG, Bailey DR, Hood DH, Coysh HL. Health checks in general practice: a comparison of two invitation letters. British Journal of General Practice. 1991;41(351):432–433. [PMC free article: PMC1371831] [PubMed: 1777303]
Ore L, Hagoel L, Shifroni G, Rennert G. Compliance with mammography screening in Israeli women: the impact of a pre-scheduled appointment and of the letter-style. Israel Journal of Medical Sciences. 1997;33(2):103–111. [PubMed: 9254871]
Pritchard DA, Straton JA, Hyndman J. Cervical screening in general practice. Australian journal of public health. 1995;19(2):167–172. [PubMed: 7786943]
Puech M, Ward J, Lajoie V. Postcard reminders from GPs for influenza vaccine: are they more effective than an ad hoc approach? Australian & New Zealand Journal of Public Health. 1998;22(2):254–256. [PubMed: 9744187]
Reda S, Makhoul S. Prompts to encourage appointment attendance for people with serious mental illness. Cochrane Database of Systematic Reviews. 2001;2 CD002085. [PMC free article: PMC7017849] [PubMed: 11406031]
Schapira DV, Kumar NB, Clark RA, Yag C. Mammography screening credit card and compliance. Cancer. 1992;70(2):509–512. [PubMed: 1617601]
Stead MJ, Wallis MG, Wheaton ME. Improving uptake in non-attenders of breast screening: selective use of second appointment. Journal of Medical Screening. 1998;5(2):69–72. [PubMed: 9718524]
Steele A. Computer telephony solution reduces no-shows. Health Management Technology. 1999;20(8):8–10. [PubMed: 10558076]
Stickney P. What works. Telephone reminder system works with patient software to reduce no-show rate. Health Management Technology. 1997;18(6):46. [PubMed: 10167520]
Taplin SH, Anderman C, Grothaus L, Curry S, Montano D. Using physician correspondence and postcard reminders to promote mammography use. American Journal of Public Health. 1994;84(4):571–574. [PMC free article: PMC1614799] [PubMed: 8154558]
Vogt TM, Glass A, Glasgow RE, La Chance PA, Lichtenstein E. The safety net: a cost-effective approach to improving breast and cervical cancer screening. Journal of Women's Health. 2003;12(8):789–798. [PubMed: 14588129]
Whittle J, Schectman G, Lu N, Baar B, Mayo-Smith MF. Relationship of scheduling interval to missed and cancelled clinic appointments. Journal of Ambulatory Care Management. 2008;31(4):290–302. [PubMed: 18806590]
Wolosin RJ. Effect of appointment scheduling and reminder postcards on adherence to mammography recommendations. Journal of family practice. 1990;30(5):542–547. [PubMed: 2332744]

INELIGIBLE STUDY DESIGN

Alaeddini A, Yang K, Reddy C, Yu S. A probabilistic model for predicting the probability of no-show in hospital appointments. Health Care Manag Sci. 2011;14(2):146–157. [PubMed: 21286819]
Anonymous. Using reminder/recall as an immunization intervention strategy works. Michigan Medicine. 2004;103(5):23. [PubMed: 15484788]
Christie J. Cochrane review brief: Email for the coordination of healthcare appointments and attendance reminders. Online Journal of Issues in Nursing. 2013;18(2):13. [PubMed: 23758431]
Cronin P, Goodall S, Lockett T, O'Keefe CM, Norman R, Church J. Cost-effectiveness of an advance notification letter to increase colorectal cancer screening. International Journal of Technology Assessment in Health Care. 2013;29(3):261–268. [PubMed: 23778152]
Greer CA. Modernizing the appointment reminder process. Journal of Medical Practice Management. 2014;30(1):67–69. [PubMed: 25241456]
Henderson R. Encouraging attendance at outpatient appointments: can we do more? Scottish Medical Journal. 2008;53(1):9–12. [PubMed: 18422203]
Lesins R. What works. Right on schedule. California healthcare organization finds an automated scheduling system that can keep pace with its increase in patient volume. Health Management Technology. 2003;24(3):44–46. [PubMed: 12647617]
McMullen MJ, Netland PA. Lead time for appointment and the no-show rate in an ophthalmology clinic. Clin Ophthalmol. 2015;9:513–516. [PMC free article: PMC4370946] [PubMed: 25834388]
Murray M, Bodenheimer T, Rittenhouse D, Grumbach K. Improving timely access to primary care: case studies of the advanced access model. JAMA. 2003;289(8):1042–1046. [PubMed: 12597761]
Ogrodniczuk JS, Joyce AS, Piper WE. Strategies for reducing patient-initiated premature termination of psychotherapy. Harvard Review of Psychiatry. 2005;13(2):57–70. [PubMed: 16020021]
Pardue S. Three quick fixes for broken appointments. Journal - Oklahoma Dental Association. 2010;101(5):24–25. [PubMed: 20806634]
Sternberg DJ. Increase your kept-appointment rate. Dental Economics. 1996;86(2):38–40. [PubMed: 9020635]
Willis BC, Ndiaye SM, Hopkins DP, Shefer A. Improving influenza, pneumococcal polysaccharide, and hepatitis B vaccination coverage among adults aged <65 years at high risk: a report on recommendations of the Task Force on Community Preventive Services. MMWR Recomm Rep. 2005;54(RR-5):1–11. [PubMed: 15800472]

INELIGIBLE INTERVENTION

Baron RC, Melillo S, Rimer BK, et al. Intervention to increase recommendation and delivery of screening for breast, cervical, and colorectal cancers by healthcare providers a systematic review of provider reminders. American Journal of Preventive Medicine. 2010;38(1):110–117. [PubMed: 20117566]
Bundy DG, Randolph GD, Murray M, Anderson J, Margolis PA. Open access in primary care: results of a North Carolina pilot project. Pediatrics. 2005;116(1):82–87. [PubMed: 15995036]
Drescher CW, Nelson J, Peacock S, Andersen MR, McIntosh MW, Urban N. Compliance of average- and intermediate-risk women to semiannual ovarian cancer screening. Cancer Epidemiology, Biomarkers & Prevention. 2004;13(4):600–606. [PubMed: 15066925]
Loo TS, Davis RB, Lipsitz LA, et al. Electronic medical record reminders and panel management to improve primary care of elderly patients. Archives of Internal Medicine. 2011;171(17):1552–1558. [PubMed: 21949163]
Pomerantz A, Cole BH, Watts BV, Weeks WB. Improving efficiency and access to mental health care: combining integrated care and advanced access. Gen Hosp Psychiatry. 2008;30(6):546–551. [PubMed: 19061681]
Rose KD, Ross JS, Horwitz LI. Advanced access scheduling outcomes: a systematic review. Archives of Internal Medicine. 2011;171(13):1150–1159. [PMC free article: PMC3154021] [PubMed: 21518935]
Shuter J, Kalkut GE, Pinon MW, Bellin EY, Zingman BS. A computerized reminder system improves compliance with Papanicolaou smear recommendations in an HIV care clinic. International Journal of STD & AIDS. 2003;14(10):675–680. [PubMed: 14596771]

INELIGIBLE OUTCOME

Baysal HY, Gozum S. Effects of health beliefs about mammography and breast cancer and telephone reminders on re-screening in Turkey. Asian Pacific Journal of Cancer Prevention: Apjcp. 2011;12(6):1445–1450. [PubMed: 22126479]
Benzel JL, Laubach PD, Griner E, et al. Improving mammography screening. American Journal of Nursing. 2009;109(11 Suppl):43–45. [PubMed: 19826337]
Makeham MA, Saltman DC, Kidd MR. Lessons from the TAPS study--recall and reminder systems. Australian Family Physician. 2008;37(11):923–924. [PubMed: 19037466]
McDermott R, Tulip F, Schmidt B, Sinha A. Sustaining better diabetes care in remote indigenous Australian communities. BMJ. 2003;327(7412):428–430. [PMC free article: PMC188495] [PubMed: 12933731]
Mukund Bahadur KC, Murray PJ. Cell phone short messaging service (SMS) for HIV/AIDS in South Africa: a literature review. Studies in Health Technology & Informatics. 2010;160(Pt 1):530–534. [PubMed: 20841743]
Patel S, Bay RC, Glick M. A systematic review of dental recall intervals and incidence of dental caries. Journal of the American Dental Association. 2010;141(5):527–539. [PubMed: 20436100]
Pereira JA, Quach S, Heidebrecht CL, et al. Barriers to the use of reminder/recall interventions for immunizations: a systematic review. BMC Medical Informatics & Decision Making. 2012;12:145. [PMC free article: PMC3541955] [PubMed: 23245381]

INELIGIBLE COMPARATOR OR NO COMPARISON

Kearins O, Walton J, O'Sullivan E, Lawrence G. Invitation management initiative to improve uptake of breast cancer screening in an urban UK Primary Care Trust. Journal of Medical Screening. 2009;16(2):81–84. [PubMed: 19564520]
Lerchenfeldt SM, Cronin SM, Chandrasekar PH. Vaccination adherence in hematopoietic stem cell transplant patients: a pilot study on the impact of vaccination cards and reminder telephone calls. Transplant Infectious Disease. 2013;15(6):634–638. [PubMed: 23890163]
Miller PL, McConnell CR, Heck JJ. Cancellations and no-shows: an examination of influences and solutions. NAHAM Management Journal. 1996;22(4):15–17. [PubMed: 10155990]
Steele RJ, Kostourou I, McClements P, et al. Effect of repeated invitations on uptake of colorectal cancer screening using faecal occult blood testing: analysis of prevalence and incidence screening. BMJ. 2010;341:c5531. [PMC free article: PMC2965320] [PubMed: 20980376]

INELIGIBLE POPULATION

Mallard SD, Leakeas T, Duncan WJ, Fleenor ME, Sinsky RJ. Same-day scheduling in a public health clinic: a pilot study. Journal of Public Health Management & Practice. 2004;10(2):148–155. [PubMed: 14967982]
O'Connor ME, Matthews BS, Gao D. Effect of open access scheduling on missed appointments, immunizations, and continuity of care for infant well-child care visits. Arch Pediatr Adolesc Med. 2006;160(9):889–893. [PubMed: 16953011]

STUDIES ON REMINDERS FOR SCREENING AND PREVENTIVE CARE

Abdullah F, Su TT. Applying the Transtheoretical Model to evaluate the effect of a call-recall program in enhancing Pap smear practice: a cluster randomized trial. Preventive Medicine. 2013;57 Suppl:S83–86. [PubMed: 23415623]
Acera A, Manresa JM, Rodriguez D, et al. Analysis of three strategies to increase screening coverage for cervical cancer in the general population of women aged 60 to 70 years: The CRICERVA study. BMC women's health. 2014;14(1) [PMC free article: PMC4106208] [PubMed: 25026889]
Brouwers MC, De Vito C, Bahirathan L, et al. What implementation interventions increase cancer screening rates? a systematic review. Implementation Science. 2011;6:111. [PMC free article: PMC3197548] [PubMed: 21958556]
Camilloni L, Ferroni E, Cendales BJ, et al. Methods to increase participation in organised screening programs: a systematic review. BMC Public Health. 2013;13:464. [PMC free article: PMC3686655] [PubMed: 23663511]
Downing SG, Cashman C, McNamee H, Penney D, Russell DB, Hellard ME. Increasing chlamydia test of re-infection rates using SMS reminders and incentives. Sexually Transmitted Infections. 2013;89(1):16–19. [PubMed: 22728911]
Everett T, Bryant A, Griffin MF, Martin-Hirsch PP, Forbes CA, Jepson RG. Interventions targeted at women to encourage the uptake of cervical screening. Cochrane Database of Systematic Reviews. 2011;5 CD002834. [PMC free article: PMC4163962] [PubMed: 21563135]
Fahey T, Schroeder K, Ebrahim S. Educational and organisational interventions used to improve the management of hypertension in primary care: a systematic review. British Journal of General Practice. 2005;55(520):875–882. [PMC free article: PMC1570766] [PubMed: 16282005]
Fortuna RJ, Idris A, Winters P, et al. Get screened: a randomized trial of the incremental benefits of reminders, recall, and outreach on cancer screening. Journal of General Internal Medicine. 2014;29(1):90–97. [PMC free article: PMC3889981] [PubMed: 24002626]
Guy R, Hocking J, Low N, et al. Interventions to increase rescreening for repeat chlamydial infection. Sexually Transmitted Diseases. 2012;39(2):136–146. [PubMed: 22249303]
Guy R, Wand H, Knight V, Kenigsberg A, Read P, McNulty AM. SMS reminders improve re-screening in women and heterosexual men with chlamydia infection at Sydney Sexual Health Centre: a before-and-after study. Sexually Transmitted Infections. 2013;89(1):11–15. [PubMed: 22517890]
Jacobson Vann JC, Szilagyi P. Patient reminder and recall systems to improve immunization rates. Cochrane Database of Systematic Reviews. 2009;(4) [PMC free article: PMC6485483] [PubMed: 16034918]
Kesman RL, Rahman AS, Lin EY, Barnitt EA, Chaudhry R. Population informatics-based system to improve osteoporosis screening in women in a primary care practice. Journal of the American Medical Informatics Association. 2010;17(2):212–216. [PMC free article: PMC3000786] [PubMed: 20190066]
Middleton P, Crowther CA. Reminder systems for women with previous gestational diabetes mellitus to increase uptake of testing for type 2 diabetes or impaired glucose tolerance. Cochrane Database of Systematic Reviews. 2014;3 CD009578. [PubMed: 24638998]
Offman J, Myles J, Ariyanayagam S, et al. A telephone reminder intervention to improve breast screening information and access. Public Health. 2014;128(11):1017–1022. [PubMed: 25443131]
Romaire MA, Bowles EJ, Anderson ML, Buist DS. Comparative effectiveness of mailed reminder letters on mammography screening compliance. Preventive Medicine. 2012;55(2):127–130. [PMC free article: PMC3694128] [PubMed: 22627089]
Stockwell MS, Westhoff C, Kharbanda EO, et al. Influenza vaccine text message reminders for urban, low-income pregnant women: a randomized controlled trial. American Journal of Public Health. 2014;104 Suppl 1:e7–12. [PMC free article: PMC4011110] [PubMed: 24354839]
Thomas RE, Lorenzetti DL. Interventions to increase influenza vaccination rates of those 60 years and older in the community. Cochrane Database of Systematic Reviews. 2014;7 CD005188. [PMC free article: PMC6464876] [PubMed: 24999919]
Vernon SW, McQueen A, Tiro JA, del Junco DJ. Interventions to promote repeat breast cancer screening with mammography: a systematic review and meta-analysis. Journal of the National Cancer Institute. 2010;102(14):1023–1039. [PMC free article: PMC2907406] [PubMed: 20587790]

EVIDENCE TABLES

DATA ABSTRACTION OF INCLUDED SYSTEMATIC REVIEWS

Download PDF (64K)

DATA ABSTRACTION OF INCLUDED PRIMARY STUDIES

QUALITY ASSESSMENT OF INCLUDED PRIMARY STUDIES

Quality Assessment of Observational Studies

Download PDF (35K)

Quality Assessment of RCTs

Download PDF (17K)

STRENGTH OF EVIDENCE FOR INCLUDED STUDIES

Strength of Evidence for KQ 1

Download PDF (16K)

Strength of Evidence for KQ2

KQ2: Reminders for Existing Appointments

Download PDF (57K)

KQ2: All Other Studies

Download PDF (17K)

PEER REVIEW COMMENT TABLE

Comment #	Reviewer #	Comment	Author Response
Are the objectives, scope, and methods for this review clearly described?
1	1	Yes	None
2	2	Yes	None
3	3	Yes	None
4	4	Yes	None
Is there any indication of bias in our synthesis of the evidence?
5	1	No	None
6	2	No	None
7	3	No	None
8	4	No	None
Are there any published or unpublished studies that we may have overlooked?
9	1	No	None
10	2	No	None
11	3	No	None
12	4	No	None
Additional suggestions or comments can be provided below.
13	1	Pg 1/Ln 19; remove “to” after (ACAP)	Edit made
14	1	Pg 2/Ln 15 - consider changing this to “in which a patient needs to be seen 3 or more months from today.” “Within” implies less than 3 months.	Changed to “in more than 90 days”.
15	1	Sometimes in the report 90 days is used and in other places 3 months. I'd suggest making it consistent throughout the report.	Revised to use “90 days” throughout the report.
16	1	Pg 2/Ln 38; - add “being” after “forgetfulness not”	Edit made
17	1	Pg 3/Ln 7; note this is unpublished data when citing reference 8.	Edit made
18	1	Pg 3/Ln 10; “remind them” not “reminder them”	Edit made
19	1	Results- Literature Flow: The number of selected studies is inconsistent. In the text, 2 studies were selected for KQ1 but Figure 1 implies 3 studies for KQ1. The text implies that none of the articles for KQ1 answer KQ2 but the footnote in Figure 1 implies some studies answer both questions.	We included 3 studies in KQ1 and clarified this in the text: “For Key Question 1, we only identified 2 flawed single-site non-concurrently controlled cohort studies that compared different approaches to scheduling follow-up appointments and one systematic review that compared different methods of scheduling initial appointments.” One systematic review addressed both KQ1 and KQ2, we clarified this in the footnote in Figure 1.
20	1	Title for KQ2 on page 12- line 12: Patient is mispelled.	Edit made
21	1	Page 14- line 18; Spelling- colposcopy is Should be colonscopy	Colposcopy is correct spelling. It is a gynecological follow-up procedure.
22	2	Page ii/Ln 27; Spelling= patient	Edit made
23	2	Pg 2/Ln 34; Question about use of “affect”	“Affect” is correct here.
24	2	Pg 3/Ln 30; add “if” to beginning of parentheses	Edit made
25	2	Pg 5/Ln 17; add “appointment” after future	Edit made
26	2	Pg 6: Analytic Framework; move wait times to intermediate outcomes and satisfaction to final outcomes; wait times and access not clearly distinguished	Moved wait times to intermediate outcomes and moved reduced satisfaction to potential consequences.
27	2	Pg 10/ Ln 59; Although appointment age would be nice to know, this wouldn't be a confounding factor because it's the principal causal pathway through which we think recall reminder reduces no-shows.	No changed needed. We agree with the reviewer's point for the comparison of 365 scheduling to recall reminder. But for the comparison of two interventions that are designed to reduce appointment age (blind scheduling close to due date vs recall reminder), knowing how well matched the appointment age is key to understanding the source of the difference; e.g., for blind scheduling, higher no-shows could also be because the patient wasn't even aware in the first place and/or didn't like the date/time, didn't like not having a say in the selection process.
28	2	Pg 11/Ln 6; Double use of word “also”	Edit made
29	2	Pg 12/ Ln 11; Spelling= patient	Edit made
30	2	Pg 12/Ln 47; question if order of percentages is correct- “Are these two comparisons in the same order? In other words is 18.2 the number with the auto reminder and is 30.9 the corresponding number for patients with depression? Although the difference is not significant, it's odd that the contrast goes in the opposite direction.”	Yes, we confirmed that the order of the percentages is correct.
31	2	Pg 17/Ln 19; Proofread this paragraph	Edited to improve clarity
32	2	Pg 18/Ln 11; Was ‘missed opportunities’ introduced and explained? If not, replace with cancellations and no-shows.	Yes, we introduced the concept of missed opportunities in the introduction.
33	2	Pg 18/Ln 18; Interesting suggestion. I don't think we have enough evidence to support a simulation model at this point.	We clarified how initiating a systems approach data collection plan could eventually inform the development of an agent-based simulation model.
34	3	The Executive summary should clearly point out that this study was a “literature search” and not a study that directly compared methods.	Added “brief evidence review” to first sentence of Executive Summary.
35	3	Pg 1/Ln 46-50; The conclusions in the example cited are erroneous and should have been discussed by the authors. i.e. using a RR to make appointments within 30 days neglects to account for the delays prior to the sending of the recall reminder. Hence, the delay is the delay from time from initial appointment (A) + time from reception of RR to actual appointment (B) Hence the delay is NOT 0 %.	No change needed. The 0% refers to proportion of patients having to wait > 30 days at time of making the follow-up appointment, not the duration of delay. Added ‘when making next appointment’ to clarify this.
36	3	Pg 2/Ln 17-21; Introduction: the purpose as described is more limited than what the “findings will drive” lines 26-29. It seems like the purpose expanded.	Edited this section to more clearly differentiate the description of the purpose of the evidence brief (i.e., to summarize the evidence on the comparative effectiveness of different approaches to scheduling follow-up appointments, lines 17-21) versus the description of how ACAP plans to use the findings in lines 26-29.
37	3	Pg 2/Ln 39; Describe reasons for missed appointment s but also include correlating factors which are not reasons- i.e. number of meds is not a reason but a correlate.	Added ‘and correlates of’
38	3	Pg 11/Ln 12; The word patient is misspelled	Edit made
39	3	Pg 18/Ln 20-30; Conclusion- I agree with this conclusion. It may be worthwhile to emphasize the individualization approach as an opportunity for future research	Changed Future Research sentence in Conclusion to be more specific about directions for future research, including individualization approach.
40	3	Pg 18/Ln 45; There is a difference in reasons for no show between new and established patients. This might be explored but the two groups are not directly comparable.	Agreed and improved the clarity of this distinction to Key Question 1's section on evidence of scheduling new patients.
41	3	This study outlined the study questions, pursued a rigorous literature search and, appropriately, could not draw many significant conclusions to directly answer the study questions. The study appropriately suggested more study. While the study questions are valid questions, the method of research- a literature search- is limited due to the lack of correlation or consistency between the study questions and the examples found in the literature. In other words, the other studies were not designed in the same way, do not investigate comparable situations, nor do the outside studies contain the same variables. As such, a literature search may not be the best way to answer these questions.	Agreed and suggested the Directions for future VA quality improvement initiatives include evaluation of (1) a complete set of pertinent and related system outcomes, (2) policy options of more flexibility and adaptation to local circumstances, (3) the impact of potential patient, provider and system effect modifiers, (4) the impact of variation in recall reminder scheduling system design (ie, how and when Veterans are contacted), (5) the independent contributions from the scheduling and reminder components, respectively, (6) the use agent-based models to identify areas with greatest potential for change, and (7) tailoring the scheduling approach to the individual Veteran.
42	4	The report found very limited evidence on comparative effectiveness of different systems for scheduling established patients' follow-up appointments. I believe that this is a true finding, and the background was comprehensive and the methodology was rigorous. I have two comments which, if addressed, would raise my recommendation from “fair” to “good”: First comment: The assumption in the report is that “missed opportunities” represent a measure of efficiency. This appears to have been the explicit instruction to the ESP CC by the DUSHOM. I recommend that consideration be given to acknowledging that a missed appointment may reflect needed care that was not delivered. Second comment: The intent of “Key Question 2” is confusing. If the intent (to differentiate the question from Key Question 1) is to focus on initial, rather than follow-up appointments, the wording should be changed to state “initial future appointments”. The content, however, that this question seems to be addressing is whether there is evidence that among patients with a scheduled future appointment, what is the comparative effectiveness of different reminder systems. This is an important question that could be helpful in designing best interventions to keep patients engaged in their care, and the finding (moderate-strength evidence) that live telephone reminders increase attendance (among patients with scheduled appointments) compared to automated telephone reminders is important, and would be worth including in the executive summary.	First comment: Refined related sentence in Background to better emphasize this point. Second comment: Both Key Questions are focused on follow-up appointments. Key Question 1 addresses overall comparative effectiveness and the purpose of Key Question 2 is to evaluate potential effect modifiers. We added findings from Key Question 2 to the executive summary.
43	4	Finally (and this is a comment that isn't at all about the quality of this ESP) The interim guidance/outpatient scheduling policy was released on May 18th. It continues to require use of recall software (with an exemption possible for sites with low missed opportunity rates) and to prohibit blind scheduling, even when there is little to no evidence for either of these strategies, as found in this document. I do hope that the current pilots will provide helpful information about potential best practices.	No change to the ESP report needed.

REFERENCES

1.: Atherton H, Sawmynaden P, Meyer B, Car J. Email for the coordination of healthcare appointments and attendance reminders. Cochrane Database of Systematic Reviews. 2012;8 CD007981. [PubMed: 22895971]
2.: Car J, Gurol-Urganci I, de Jongh T, Vodopivec-Jamsek V, Atun R. Mobile phone messaging reminders for attendance at healthcare appointments. Cochrane Database of Systematic Reviews. 2012;7 CD007458. [PubMed: 22786507]
3.: Free C, Phillips G, Watson L, et al. The effectiveness of mobile-health technologies to improve health care service delivery processes: a systematic review and meta-analysis. PLoS Medicine / Public Library of Science. 2013;10(1):e1001363. [PMC free article: PMC3566926] [PubMed: 23458994]
4.: George A, Rubin G. Non-attendance in general practice: a systematic review and its implications for access to primary health care. Family Practice. 2003;20(2):178–184. [PubMed: 12651793]
5.: GurolUrganci I, de Jongh T, VodopivecJamsek V, Atun R, Car J. Mobile phone messaging reminders for attendance at healthcare appointments. Cochrane Database of Systematic Reviews. 2013;(12) [PMC free article: PMC6485985] [PubMed: 24310741]
6.: Guy R, Hocking J, Wand H, Stott S, Ali H, Kaldor J. How effective are short message service reminders at increasing clinic attendance? A meta-analysis and systematic review. Health Services Research. 2012;47(2):614–632. [PMC free article: PMC3419880] [PubMed: 22091980]
7.: Hasvold PE, Wootton R. Use of telephone and SMS reminders to improve attendance at hospital appointments: a systematic review. Journal of Telemedicine & Telecare. 2011;17(7):358–364. [PMC free article: PMC3188816] [PubMed: 21933898]
8.: Liu Q, Abba K, Alejandria MM, Sinclair D, Balanag VM, Lansang MA. Reminder systems to improve patient adherence to tuberculosis clinic appointments for diagnosis and treatment. Cochrane Database of Systematic Reviews. 2014;11 CD006594. [PMC free article: PMC4448217] [PubMed: 25403701]
9.: McLean S, Gee M, Booth A, et al. Targeting the Use of Reminders and Notifications for Uptake by Populations (TURNUP): a systematic review and evidence synthesis. National Institute for Health Research; Southampton, UK: 2014. [PubMed: 25642537]
10.: Reda S, Rowett M, Makhoul S. Prompts to encourage appointment attendance for people with serious mental illness. Cochrane Database of Systematic Reviews. 2012;(8)
11.: Schauman O, Aschan LE, Arias N, Beards S, Clement S. Interventions to increase initial appointment attendance in mental health services: a systematic review. Psychiatric Services. 2013;64(12):1249–1258. [PubMed: 24036532]
12.: Stubbs ND, Geraci SA, Stephenson PL, Jones DB, Sanders S. Methods to reduce outpatient non-attendance. American Journal of the Medical Sciences. 2012;344(3):211–219. [PubMed: 22475731]
13.: Brannan SO, Dewar C, Taggerty L, Clark S. The effect of short messaging service text on non-attendance in a general ophthalmology clinic. Scottish Medical Journal. 2011;56(3):148–150. [PubMed: 21873719]
14.: Cherniack EP, Sandals L, Gillespie D, Maymi E, Aguilar E. The use of open-access scheduling for the elderly. Journal for Healthcare Quality. 2007;29(6):45–48. [PubMed: 18232607]
15.: Farmer T, Brook G, McSorley J, Murphy S, Mohamed A. Using short message service text reminders to reduce ‘did not attend’ rates in sexual health and HIV appointment clinics. International Journal of STD & AIDS. 2014;25(4):289–293. [PubMed: 23999939]
16.: Haufler K, Harrington M. Using nurse-to-patient telephone calls to reduce day-of-surgery cancellations. AORN Journal. 2011;94(1):19–26. [PubMed: 21722768]
17.: Henry SR, Goetz MB, Asch SM. The effect of automated telephone appointment reminders on HIV primary care no-shows by veterans. Journal of the Association of Nurses in AIDS Care. 2012;23(5):409–418. [PubMed: 22424961]
18.: McInnes DK, Petrakis BA, Gifford AL, et al. Retaining homeless veterans in outpatient care: a pilot study of mobile phone text message appointment reminders. American Journal of Public Health. 2014;104 Suppl 4:S588–594. [PMC free article: PMC4151902] [PubMed: 25100425]
19.: Perry JG. A preliminary investigation into the effect of the use of the Short Message Service (SMS) on patient attendance at an NHS Dental Access Centre in Scotland. Primary Dental Care. 2011;18(4):145–149. [PubMed: 21968040]
20.: Saine PJ, Baker SM. What is the best way to schedule patient follow-up appointments? Joint Commission Journal on Quality & Safety. 2003;29(6):309–315. [PubMed: 14564749]
21.: Parikh A, Gupta K, Wilson AC, Fields K, Cosgrove NM, Kostis JB. The effectiveness of outpatient appointment reminder systems in reducing no-show rates. American Journal of Medicine. 2010;123(6):542–548. [PubMed: 20569761]
22.: Junod Perron N, Dao MD, Kossovsky MP, et al. Reduction of missed appointments at an urban primary care clinic: a randomised controlled study. BMC Family Practice. 2010;11:79. [PMC free article: PMC2984453] [PubMed: 20973950]
23.: Junod Perron N, Dao MD, Righini NC, et al. Text-messaging versus telephone reminders to reduce missed appointments in an academic primary care clinic: a randomized controlled trial. BMC Health Services Research. 2013;13:125. [PMC free article: PMC3623700] [PubMed: 23557331]
24.: Taylor NF, Bottrell J, Lawler K, Benjamin D. Mobile telephone short message service reminders can reduce nonattendance in physical therapy outpatient clinics: a randomized controlled trial. Archives of Physical Medicine & Rehabilitation. 2012;93(1):21–26. [PubMed: 22000821]
25.: Macharia WM, Leon G, Rowe BH, Stephenson BJ, Haynes RB. An overview of interventions to improve compliance with appointment keeping for medical services. JAMA. 1992;267(13):1813–1817. [PubMed: 1532036]

Prepared for: Department of Veterans Affairs, Veterans Health Administration, Quality Enhancement Research Initiative, Health Services Research & Development Service, Washington, DC 20420

Prepared by: Evidence-based Synthesis Program (ESP), Coordinating Center, Portland VA Medical Center, Portland, OR, Mark Helfand, MD, MPH, MS, Director

Recommended citation: Peterson K, McCleery E, Anderson J, Waldrip K, Helfand M. Evidence Brief: Comparative Effectiveness of Appointment Recall Reminder Procedures for Follow-up Appointments. VA ESP Project #09-199; 2015 [PubMed: 27606388].

This report is based on research conducted by the Evidence-based Synthesis Program (ESP) Coordinating Center located at the Portland VA Health Care System, Portland, OR, funded by the Department of Veterans Affairs, Veterans Health Administration, Office of Research and Development, Quality Enhancement Research Initiative. The findings and conclusions in this document are those of the author(s) who are responsible for its contents; the findings and conclusions do not necessarily represent the views of the Department of Veterans Affairs or the United States government. Therefore, no statement in this article should be construed as an official position of the Department of Veterans Affairs. No investigators have any affiliations or financial involvement (eg, employment, consultancies, honoraria, stock ownership or options, expert testimony, grants or patents received or pending, or royalties) that conflict with material presented in the report.

Created: July 2015.

Bookshelf ID: NBK384609PMID: 27606388

PubReader
Print View
Cite this Page
Peterson K, McCleery E, Anderson J, et al. Evidence Brief: Comparative Effectiveness of Appointment Recall Reminder Procedures for Follow-up Appointments [Internet]. Washington (DC): Department of Veterans Affairs (US); 2015 Jul.
PDF version of this title (562K)

Other titles in this collection

VA Evidence Synthesis Program Reports

Related information

NLM Catalog
Related NLM Catalog Entries
PMC
PubMed Central citations
PubMed
Links to PubMed

Recent Activity

Clear Turn Off Turn On

Evidence Brief: Comparative Effectiveness of Appointment Recall Reminder Procedu...
Evidence Brief: Comparative Effectiveness of Appointment Recall Reminder Procedures for Follow-up Appointments

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

Bookshelf

Evidence Brief: Comparative Effectiveness of Appointment Recall Reminder Procedures for Follow-up Appointments

PREFACE

EXECUTIVE SUMMARY

Table A

Background

Methods

INTRODUCTION

PURPOSE

BACKGROUND

SCOPE

KEY QUESTIONS

ELIGIBILITY CRITERIA

ANALYTIC FRAMEWORK

Figure 1

METHODS

RESULTS

LITERATURE FLOW

Figure 2

Table 1

KEY QUESTION 1. For adult patients who are targeted for follow-up appointments, what is the comparative effectiveness of appointment Recall Reminder (RR) procedures versus other kinds of follow-up appointment scheduling systems?

Comparison of different methods for follow-up patient scheduling

Comparison of different methods for new patient scheduling

Blind scheduling versus patient-initiated

Accelerated access

A. Patient factors

A1. New versus established patients

A2. Patient clinical factors

A3. Patient preference

A4. Patient age

A5. Other patient factors

B. Appointment scheduling systems engineering design and management factors

B1. Mode of notification

B2. Threshold for notification (1 month, 2 weeks)

B3. Mode of patient response

B4. Reminder mode

Table 2

C. Facility characteristics

SUMMARY AND DISCUSSION

LIMITATIONS

FUTURE RESEARCH

Table 3

CONCLUSIONS

REFERENCES

SUPPLEMENTAL MATERIALS

SEARCH STRATEGIES

SYSTEMATIC REVIEWS

Database: Ovid MEDLINE (April 7, 2015)

Database: Cochrane Database of Systematic Reviews (March 5, 2015)

PRIMARY STUDIES

Database: Ovid MEDLINE and Cochrane Central Registry of Controlled Trials (March 5, 2015)

LIST OF EXCLUDED STUDIES

PRIMARY STUDIES ON REMINDERS FOR EXISTING APPOINTMENTS PUBLISHED BEFORE 2010

INELIGIBLE STUDY DESIGN

INELIGIBLE INTERVENTION

INELIGIBLE OUTCOME

INELIGIBLE COMPARATOR OR NO COMPARISON

INELIGIBLE POPULATION

STUDIES ON REMINDERS FOR SCREENING AND PREVENTIVE CARE

EVIDENCE TABLES

DATA ABSTRACTION OF INCLUDED SYSTEMATIC REVIEWS

DATA ABSTRACTION OF INCLUDED PRIMARY STUDIES

Data Abstraction of Observational Studies

Data Abstraction of RCTs

QUALITY ASSESSMENT OF INCLUDED SYSTEMATIC REVIEWS

QUALITY ASSESSMENT OF INCLUDED PRIMARY STUDIES

Quality Assessment of Observational Studies

Quality Assessment of RCTs

STRENGTH OF EVIDENCE FOR INCLUDED STUDIES

Strength of Evidence for KQ 1

Strength of Evidence for KQ2

KQ2: Reminders for Existing Appointments

KQ2: All Other Studies

PEER REVIEW COMMENT TABLE

REFERENCES

Views

In this Page

Other titles in this collection

Related information

Similar articles in PubMed