SUMMARY

There exist a solid evidence base for quality domains and an extensive list of potential indicators that can be used to measure quality performance in family planning programs. The Family Planning Annual Report (FPAR), the Family Planning Council of America Performance Monitoring System (FPCA), and Healthy People 2010 reproductive health goals are explicitly specified indicators representative of the more than 200 indicators that have been suggested in this arena. There is some consistency (or overlap) in indicators among these documents, and several of the indicators reflect goals adopted by external agencies, such as the Healthcare Employer Data and Information Set (HEDIS) measures espoused by the National Center for Quality Assurance (NCQA) (specifically breast and cervical cancer screening and screening for chlamydia). Two obvious deficiencies in the currently reported measures are:

• The lack of outcome data that are patient-specific about reproductive desires (specifically Helping patients Achieve their Reproductive Intentions [HARI]; patients should plan for pregnancy as well as plan to prevent pregnancy).
• The lack of data on provider competency and interpersonal skills or client comprehension/literacy.

The Title X program mandate specifies three long-term measures that are to be reported annually: (1) increasing the number of unintended preg nancies averted by providing Title X family planning services; (2) reducing infertility among women attending family planning clinics by identifying chlamydia infection; and (3) reducing invasive cervical cancer among women attending family planning clinics. Additional pertinent indicators include screening for other sexually transmitted diseases (STDs) that are treatable and preventable and have significant maternal and perinatal long-term implications (syphilis, gonorrhea, HIV). Available monitoring systems adequately address these indicators.

Future primary data collection efforts should include the following patient-centered priorities:

• Patient-specific reproductive desires/outcomes. Instead of counting visits and number of new visits, the focus should be changed to:
  • HARI: What are pregnancy plans for the year? Among those making a repeat visit, have these goals been met?
  • Percent clients not pregnant at next visit (denominator: those planning contraception)
  • Percent clients still using any method
  • Percent referrals for pregnancy termination or percent referrals for prenatal care for unintended pregnancy
  • Percent pregnant who desired pregnancy

• Patient-specific evaluation of the quality of information provided
  • Technical competence and interpersonal skills of provider
  • Client comprehension (health literacy)

This appendix addresses the measurement of the quality of reproductive health services provided under the Title X program. The discussion includes an assessment of how well the FPAR measures quality, a description of quality initiatives undertaken by family planning programs, and consideration of how the quality of services should be assessed in various settings. An assessment of the costs and benefits associated with introducing quality measures into family planning clinics is beyond the scope of this discussion because of the limited data available to inform such an assessment.

OVERVIEW OF FINDINGS AND RECOMMENDATIONS

To assess quality, there must be consensus on what quality is (e.g., how it is defined), as well as agreement on what measures are to be used to monitor and report quality. Several definitions of quality are pertinent, including those of the Institute of Medicine (IOM) and the World Health Organization (WHO) (WHO, 1998; IOM, 2001). Both emphasize proper performance of care based on current standards and knowledge, recognizing the potential for individual and societal benefit. Judith Bruce offers a family planning–specific definition of quality: “providing a range of services that are safe, effective, and that satisfy clients’ needs and wants” (Bruce, 1990). For this study, the author performed a focused review of the literature and evaluated the FPARs for 2001–2006 (Frost, 2001, 2002, 2003; Frost and Frohwirth, 2005; Fowler et al., 2006; RTI International, 2006), the Title X Program Assessment Rating Tool (PART) evaluation (OMB, 2005), and an advance copy of the FPCA proposed Performance Measurement System (FPCA, 1999).

This review led to the conclusion that a full assessment of the quality of the Title X program cannot be performed at this time. Based on the limited information available, primarily the 2005–2006 FPARs, the program does appear to be doing what it set out to do; however, the extent to which its services are underused, overused, or used inappropriately (measures of poor quality) cannot be determined from these reports. Whereas the readily apparent structure and process variables appear to have face and construct validity, the outputs and outcomes need further clarification. Regional and/or population data are needed to support any claim for program effects. Think tanks and advocacy groups, such as the Alan Guttmacher Institute and the Center for Reproductive Rights, believe there is a logic model to support a causal link between family planning services and pregnancies averted and dollars saved (Center for Reproductive Rights, 2004; Dreweke, 2006). Evidence of such program impact would clearly help advance the policy mandate for more funding, more marketing, and the development of more meaningful indicators to advance the reproductive health agenda.

While there are substantial data to support a framework for both quality assessment and program evaluation within the family planning field, there are limited data on the quality of national family planning services, and there does not appear to be a national consensus about the quality domains or quality indicators that should be routinely (or periodically) monitored and reported. There is a tendency to count resources, visits, and tests, with less energy directed toward capturing data on intermediate effects or long-term impact, such as pregnancies prevented (or planned) or overall reduction in population fertility rates or STD rates.

Surprisingly, the lack of data on the quality of family planning services in the United States in general and under Title X in particular is not due to the lack of an evidence base for indicators, but to an apparent failure to capitalize on the extensive work that has been done and applied internationally in this arena. Similarly, and not surprisingly, the dearth of quality-of-care research in the area of family planning in the United States contrasts with the quality-of-care work in the medical/surgical arena, likely because family planning has historically been focused primarily on women. Despite widespread acceptance internationally, only recently have the benefits of family planning been claimed to extend to improved child, family, and world health in the United States (Cleland et al., 2006). The inclusion of reproductive health services that encompass infertility and STD/HIV screening and treatment has broadened family planning services to extend to outreach programs for men, but this, too, has been a relatively new phenomenon. The absence or relative paucity of indicators for women’s health, maternity services, and child health has not gone unnoticed by health service researchers, but progress on the development of these indicators has been slow (Schuster et al., 1997; Kerr et al., 2000; Gregory et al., 2005; Korst et al., 2005). In fact, the Agency for Healthcare Research and Quality (AHRQ)—the national leader in advancing the quality agenda—specifically excludes pregnancy and children from its current inpatient and patient safety indicators (AHRQ, 2004, 2006a,b). A set of pediatric inpatient indicators was recently developed (AHRQ, 2006c).

Attempts to achieve federal accountability across all federally funded programs have spurred the development of indicators for Title X. Current efforts by the FPCA to develop consensus-based performance indicators are a step in the right direction. Efforts to capture additional measures, already defined by the Department of Health and Human Services (HHS) via Healthy People 2010, would further the cause (HHS, 2000a). Examples of representative Healthy People 2010 reproductive and STD/HIV goals that would be consistent with Title X program goals can be found in Annex K-1. Similarly, focused incorporation of selected indicators from the Handbook of Indicators for Family Planning Program Evaluation, which contains more than 200 indicators, would be beneficial and could elevate family planning and preventive reproductive health services in general, and the Title X program in particular, from a relatively obscure program for the poor to a more prominent national program dedicated to improving the health and well-being of women, children, and families (Bertrand et al., 1994). Glasier et al., in an editorial about family planning services and women’s health, state that “unsafe sex is the 2nd most important risk factor for disability and death in the world’s poorest communities, and the 9th most important in developed countries” (Glasier et al., 2006). These authors contend that reproductive health services are of poor quality and underused because discussions about sexual intercourse and sexuality make people uncomfortable. Further, they suggest that the increasing influence of conservative, political, religious, and cultural forces threatens to undermine what has been achieved to date.

To make this admonishment meaningful in a different social context, the average youth watches 3 hours of television daily, whereas 59 percent of adults watch television 2 or more hours (Roberts et al., 1999; Bowman, 2006). It is inevitable that most Americans will therefore encounter sexual messages given how commonly they appear on television; approximately 64 percent of all programs have sexual content based on analysis of the 2001–2002 TV season. Among programs with sexual content, there was an average of 4.4 scenes per hour. Talk about sex was more common (61 percent) than overt portrayals of sexual behaviors (32 percent). However, approximately 14 percent (or one of every seven programs) included a portrayal of sexual intercourse, depicted or strongly implied (Kunkel et al., 2003; Collins et al., 2004). Since children tend to model what they see, it is perhaps not surprising that the average age of sexual debut overall is 14.2 years (13.1 for boys, 15.0 for girls) (Sandfort et al., 2008). Moreover, the rate of premarital sex continues to be high. Fully 90 percent of women aged 15–44 responding to the National Survey of Family Growth (NSFG) in 2002 had had premarital intercourse (Mosher et al., 2004). The social marketing of sexuality is rampant, while the marketing of abstinence, contraception, and preventive reproductive behaviors is glaringly absent, despite evidence that it could be beneficial (Piotrow et al., 1997).

Based on a review of the literature and a synthesis of both national and international data regarding quality measurement in family planning, there is a solid evidence base for the quality domains that should be included, and there is no dearth of potential indicators that could be used. The FPCA Performance Measurement System, coupled with the Healthy People 2010 goals, would be an excellent start. These should be fortified with a paradigm policy shift that emphasizes planning for pregnancy as much as planning to prevent pregnancy (HARI) (Jain et al., 1992). This is entirely consistent with the IOM recommendation to adopt a social norm whereby all pregnancies are intended—clearly and consciously desired at the time of conception (IOM, 1995). Further analysis of the merits of the Title X program is needed using various methodologies, such as provider observation or simulated patients, to document technical competence and communication skills. Additionally, patient exit interviews should not be limited to satisfaction surveys or closed questions about what was discussed as is currently the trend, but should include measures verifying literacy and comprehension. Community focus groups should ascertain additional perceptions of clinic quality with less possibility of courtesy bias (Sullivan and Bertrand, 2000).

The infrastructure to implement many of these suggestions exists, as evidenced by the FPARs, the Program Guidelines, application criteria, and the Office of Population Affairs’ (OPA) 2006 Family Planning Program Priorities, Legislative Mandates, Key Issues (HHS, 2006). However, data collection efforts by Title X clinic sites may already be burdensome, and future funding should support either quality-monitoring full-time equivalents (FTEs) or an electronic medical record (EMR) system without sacrificing support for existing services. The EMRs should be designed to capture indicator data. There should be a vision or capacity for shared information (regional health information network), given the transient nature of the poor population. The opportunity for shared patient-level data (with appropriate consent and Health Insurance Portability and Accountability Act [HIPAA] considerations) could optimize STD/HIV prevention and treatment. Additionally, a widely acknowledged strength of the Title X program is its information–education–counseling (IEC) emphasis and the associated training and audiovisual tools directed at both clients and providers. These tools, coupled with standardized protocols, could be shared across state and federal agencies, including STD clinics and public schools, as well as incorporated into medical, nursing, and residency training curricula. There is clearly stakeholder support for enhancing the quality of family planning services at the user/provider level, but broader support at the administrative and policy levels is needed to facilitate the development of a national agenda emphasizing the maternal, child, and family benefits of family planning and preventive health services in general and the Title X program in particular.

The remainder of this appendix provides an overview of the theoretical frameworks that inform this review. It also provides an assessment of how well the FPAR measures quality based on these frameworks, a description of quality initiatives undertaken by family planning programs, and an overview of how the quality of services should be assessed. The latter overview uses representative examples of available indicators, highlighting where existing indicators may need to be modified or expanded to address clients’ needs and wants and the HARI principle.

THEORETICAL FRAMEWORKS

Several quality and reproductive health frameworks inform this discussion:

• Donabedian quality model: Structure–Process–Outcome Model (Donabedian, 1968)
• Bertrand et al. program evaluation model (Bertrand et al., 1994; Sullivan and Bertrand, 2000)
• International frameworks for quality family planning services
  • Bruce and Jain et al. model of quality family planning services (Bruce, 1990; Jain et al., 1992) International Planned Parenthood Federation (IPPF) framework (IPPF, 1998)
  • 1994 International Conference on Population and Development, Cairo (United Nations, 1994)

• Lu and Halfon Reproductive Health Continuum (Lu and Halfon, 2003)
• AHRQ criteria for an acceptable indicator (AHRQ, 2001)

These frameworks need to be integrated and ultimately accepted by various stakeholders.

Donabedian’s Structure–Process–Outcome Model

Donabedian’s model has been widely endorsed as the theoretical framework for quality measurement (Donabedian, 1968). Key representative variables critical for assessment of the quality of Title X programs using the various domains outlined by Donabedian are shown in Figures K-1 through K-3.

FIGURE K-1

Examples of variables representing Donabedian’s structure quality domain.

FIGURE K-3

Similarities between quality monitoring and program evaluation.

Structure

Where is the facility located? What are the physical and administrative barriers to accessing services? Is the site geographically convenient and accessible by public transportation? Are the hours varied enough to meet the needs of clients—for example, evening or weekend hours for people who work or teens in school? Are the building and waiting area physically appealing? Once inside, is the waiting area comfortable, are there enough chairs, and are there diversions for accompanying children? Is there an opportunity for private discussion between clients and various clinical and nonclinical staff? Both national and international studies evaluating client satisfaction have demonstrated that all of these factors may influence whether a client comes for an initial visit or returns for subsequent visits—which of necessity impacts the short- and long-term goals of initiation and continuation of contraceptive services (Alden, 2004; Zaky et al., 2007). These may be significant issues at the individual clinical sites, determining whether or not a client seeks initial or follow-up services. However, at the program level, only privacy issues are relevant from a regulatory standpoint. Staffing mix and training opportunities determine technical competence. The sociopolitical and cultural milieu can impact what methods are available (e.g., termination or emergency contraception). Ready access to information technology (IT) support can facilitate notification after positive test results and timely data acquisition for audits. The strength of the advisory board can determine additional funding options for outreach, social marketing, and facility improvement.

Process

The services that must be provided by Title X programs have been mandated by legislation (Figure K-2) (HHS, 2001, 2006). The federal Family Planning Program is authorized under Title X of the Public Health Service Act, which was created in 1970 to provide family planning and related preventive health care. The mission of Title X is to provide the information and means necessary for individuals to exercise choice in determining the number and spacing of their children (AHRQ, 2006c). In addition to providing a broad range of contraceptive supplies, counseling, and information on a confidential basis, clinics with Title X funding must provide reproductive health and preventive health services that include breast and pelvic exams to screen for breast cancer, cervical cancer, and STDs (including HIV); pregnancy diagnosis; patient education; reproductive health counseling; and appropriate social and referral services. A mechanism to determine that the information provided has been understood should be established and documented. Further, the Title X legislation authorizes funding for family planning services, training, research, information, and education. The program is administered by OPA through the Office of Family Planning (OFP). Services are intended for all who want and need them, with priority for low-income individuals. The Program Guidelines stipulate the involvement of an advisory board and community participation in the development of educational materials and project promotion.

FIGURE K-2

Services that must be provided by Title X programs.

While these are nationally legislated mandates, how these mandates get carried out is influenced by regional or local implementation strategies, largely resulting in site-specific policies and procedures (see Figure K-3). This variation is due to previously mentioned structural variables such as sociocultural or political milieus, regional demographics and/or case mix, and availability of staff and legal or licensing mandates regarding what types of providers can perform which types of services. For example, Gillian et al. found improved compliance among African American teens due to the involvement of all clinical staff in the process. All clinic employees, including clerical and professional staff, forged relationships through shared backgrounds and experiences, honesty, and additional time spent with the teens (Gilliam et al., 2007).

Outcome

See Figure K-3 and the discussion below of the Bertrand et al. (1994) program evaluation model.

Relationship Between Quality Assessment and Program Evaluation

It is difficult if not impossible to distinguish quality assessments from program evaluations when reviewing the family planning literature. Hence, Figure K-3 shows an integrated model of quality and program evaluation. Inputs and processes are conceptually equivalent to Donabedian’s structure and process variables. Input examples pertinent to family planning include such items as personnel, financial resources, facilities, and equipment. Both output and outcomes are comparable to Donabedian’s outcome domain. However, output specifically refers to outcomes at the program level and is usually defined by service utilization (number of visits, number of new or continuing contraceptive users). On the other hand, program outcomes defined by program effect and program impact are measured at the population level. An example of program effect is the prevalence of contraceptive use as measured in a population survey, and an example of program impact is the regional or national fertility rate or desired pregnancy rate. The maturity of the program determines the type of evaluation strategy to use—the more mature the program, the more impact one would expect to be able to demonstrate. By most standards, Title X is a mature program (more than 30 years old), but it is immature in quality assessment; hence the majority of indicators will initially be focused on processes and outputs. However, the program has been in existence long enough to be capable of demonstrating long-term impact if the correct data are made available for collection and interpretation.

International Frameworks for Quality Family Planning Services

Several authors have published extensively on quality assessment as it relates to the international family planning and reproductive health care arena. Most authors build on or adapt the Bruce (1990) and Jain et al. (1992) framework, which identifies six elements of quality:

• Choice of method,
• Information provided to the client,
• Technical competence of providers,
• Interpersonal relations between clients and providers,
• Mechanisms to encourage continuity of care, and
• Appropriate constellation of services.

Outcome measures include program readiness, the provider perspective, the services delivered, and the client perspective and the services received, with particular emphasis on client knowledge, client satisfaction, client health, and contraceptive use—both acceptance and continuation.

IPPF adopted the above framework as its model for quality after adding client acceptability to the above list and emphasizing that these criteria should be defined as clients’ and providers’ rights and expectations (IPPF, 1998). Table K-1 outlines the IPPF client and provider “bill of rights.”

TABLE K-1

International Planned Parenthood Federation Framework: Clients’ Rights and Providers’ Needs.

The most extensive and comprehensive framework for family planning quality indicators and program evaluation is that proposed by Bertrand et al. (1994). While intended for developing countries, it is easily applicable to the United States and includes both program-and population-level indicators. Program-based or performance indicators include those factors related to inputs, processes, and outputs, whereas population-based or outcome indicators usually reflect intermediate effect or long-term impact. Bertrand et al. define eight broad categories for program evaluation, which outline the pathways by which programs achieve impact in a given country:

• Indicators to measure the policy environment,
• Indicators to measure service delivery operations,
• Indicators to measure family planning outputs,
• Indicators to measure demand for children,
• Indicators to measure demand for family planning,
• Indicators to measure service utilization,
• Indicators to measure contraceptive practice, and
• Indicators to measure fertility impact.

The categories outline the pathways by which programs achieve impact in a given country (see Figures K-4 and K-5). Bertrand et al. (1994) advise that a key prerequisite for the development of a successful family planning program is the presence of a strong, viable political and administrative system. Political support should be coupled with resource allocation and permissive legal codes and regulations that affect the number, type, and distribution of methods. A comprehensive family planning service delivery program will have defined functional areas that include management, supervision, training, commodities and logistics, IEC, and evaluation. There are two primary expected results (or outputs) for a successful program: (1) adequate services from the client perspective, as defined by accessibility, quality, and acceptability; and (2) increased utilization of services, as defined by number of new and/or continuing users.

FIGURE K-4

Conceptual framework of family planning demand and program impact on fertility. SOURCE: Bertrand, J. T., R. J. Magnani, and J. Knowles. 1994. Handbook of indicators for family planning. Chapel Hill, NC: MEASURE Evaluation, Carolina Population Center. (more...)

FIGURE K-5

Conceptual framework of family planning demand and program impact on fertility. SOURCE: Bertrand, J. T., R. J. Magnani, and J. Knowles. 1994. Handbook of indicators for family planning. Chapel Hill, NC: MEASURE Evaluation, Carolina Population Center. (more...)

Each category has a series of indicators. A detailed listing can be found in Annex K-2. It should be noted that there are approximately 200 discrete descriptive indicators that can provide trend data over time; however, they are meant to be used selectively depending on what outputs, processes, or outcomes are being assessed. In fact, field research used a modified “quick” 25 indicators with good success (Sullivan and Bertrand, 2000). The indicators and data collection method are listed in Table K-2. The research focused on indicators of quality that were related to client behaviors. It was aimed at determining the feasibility of data collection, testing the comparability of results obtained with two separate instruments (direct observation of provider and client exit interview), and determining the cost of data collection, among other things. It is among the few studies that provide information on the cost of quality assessment, and also demonstrate how the data could be used and compared within and among health systems or countries. The actual dollar amount for the costs associated with conducting the studies is difficult to interpret given the lack of comparability among international salary ranges. It is noteworthy, however, that the researchers interpreted the costs as reasonable, but they were considered high by the service providers, who were concerned that those funds might need to be diverted from other family planning–related resources (range $217–1,000/site).

TABLE K-2

Short List of Quality Indicators and Method of Data Collection.

All of the above frameworks essentially culminated in a paradigm shift away from fertility control in favor of individualized services responsive to client needs. This human rights approach was widely endorsed in 1994 at the International Conference on Population and Development in Cairo (United Nations, 1994).

Recently, assessment of family planning program quality has begun to emphasize client satisfaction. WHO has specified that clients want respect, understanding, individualized care, complete and accurate information, technical competence, access, fairness, and results (WHO, 1998). Communication standards have been emphasized internationally. Clients should be given sufficient information and counseling to be capable of making contraceptive decisions (WHO, 2004). This information should include at a minimum:

• A description and understanding of the relative effectiveness of the chosen method,
• Correct use of the method,
• How it works,
• Common side effects,
• Health risks and benefits of the method,
• Signs and symptoms that should necessitate a return to the office,
• Information on return to fertility after stopping the method, and
• Information on STD protection.

It should be noted that exit interviews on site and focused interviews in the community may reveal different pictures of quality, raising concern about the validity of exit interviews and the possibility of a positive or courtesy response bias on site (Sullivan and Bertrand, 2000; RamaoRao and Mohanam, 2003).

There has been considerable research on concepts, frameworks, measurement, and methodology, but few experimental studies have been conducted on the impact of quality services on reproductive health outcomes. Few quality interventions have been designed within a research framework capable of measuring or quantifying their effects. In a review of 15 international studies, most had methodological flaws (RamaoRao and Mohanam, 2003). Proposed explanations for these limitations included a perception that research activities were outside the normal scope of work and were burdensome and consumed limited resources. The small number of well- designed intervention studies suggests that quality can be improved and that good care has beneficial effects. The authors conclude that conceptual frameworks to examine quality have been developed and refined. The language and vocabulary used to define them have been accepted by diverse constituents. Methodological advances have been developed for data collection. A variety of interventions have been tested. Intervention tools demonstrating the most promise are those that facilitate better interaction between clients and providers through provider training in interpersonal communication and information exchange or the use of audiovisual aids. Better care has been associated with higher levels of client satisfaction and contraceptive adoption and continuation. Despite these findings, the authors acknowledge a number of gaps in the literature. These gaps should serve as stimulus for discussion when considering the development of quality indicators in the United States, where indicator development is in its infancy, but program infrastructure has been well established:

• What is the impact of decentralization on the quality of care provided or received?
• What is the level of readiness and quality of care in the private sector? Studies suggest private-sector facilities do not necessarily provide better care.
• Why do family planning clients choose to use some facilities rather than others? Is choice guided by perceptions of quality, and if so, how?
• What changes would encourage clients to continue to visit facilities and stay with existing programs? Research designs and program evaluations tend to concentrate on new users rather than the needs of existing clients. What quality attributes predict return visits or behavioral changes that support birth spacing/limiting and STD prevention?
• Many indicators have been proposed, but none have been tested on a wide scale. What is the most efficient mechanism to study new measures? Can existing infrastructure support indicator development and validation?
• Can quality be improved without extraordinary financial outlay? There is limited information on costs, cost-effectiveness, and financing.

To summarize, the international literature has established the following tenets as the elements essential for quality family planning services:

• A client-centered perspective sets the program objectives and standards.
• Interactions are information based and participatory, and allow for collaborative decision making.
• Clients have access and are treated with respect, understanding, and fairness. Clients want to be given complete and accurate information and be treated by technically competent providers (RamaoRao and Mohanam, 2003).

HOW WELL DOES THE TITLE X FPAR MEASURE QUALITY?

WHAT QUALITY INITIATIVES HAVE BEEN UNDERTAKEN BY FAMILY PLANNING PROGRAMS?

This section starts with a broad overview of the utilization of family planning services and quality assessment studies of family planning programs in the United States. It then reviews what is known about the quality initiatives that have been undertaken by Title X programs, as evidenced by the PART and the proposed FPCA Performance Measurement System (FPCA, 1999; OMB, 2005).

HOW SHOULD THE QUALITY OF TITLE X SERVICES BE ASSESSED IN VARIOUS SETTINGS?

A review of the literature reveals that a theoretical framework and a multitude of quality indicators exist that can be used to assess the quality of family planning and reproductive health services in the Title X program. One can think of the indicators presented as a pyramid becoming more complex and more comprehensive as one approaches the base (see Figure K-6). At the top of the pyramid are the PART performance measures. The existing FPARs would be the next rung, but currently represent primarily descriptive indicators of structure and processes. The FPCA PMS, with its 24 indicators, will provide a mechanism to collect outputs that will yield some data on short-term program effect. Ultimately, adoption of the Healthy People 2010 reproductive health and STD benchmarks would demonstrate more significant program impact across large regions of the United States (see Appendix K-1) (HHS, 2009). Finally, individualization by site or region and selection of key representative indicators from the more than 200 indicators in the Handbook of Indicators (Bertrand et al., 1994) would help spur site-specific goals directed at quality improvement. Thus, deciding which indicators to use, how many to use, and across what settings will be a function of the goal of the evaluation. As suggested by Bertrand et al. (1994), if the goal is to advance political support, indicators showing the program effectiveness and selected indicators from the policy environment will be monitored and distributed among stakeholders who can influence policy. Small new sites might want to focus on needs assess ments, increased volume, program revenues, technical competence, and patient satisfaction. Large, established, older sites or networks might want to emphasize networking and technical training to keep staff motivated and interested. Sites frequented primarily by teens or clients with LEP would obviously have different quality improvement interests than sites with more diverse populations.

FIGURE K-6

Integration of quality assurance (program) and quality improvement (site) activities.

More rapid improvement could be achieved if national measures were established for all sites to report, with recognized benchmarks or standards. These objective measures would represent quality assurance activities coming from the top down and would serve to demonstrate both short- and long-term impact over time. Additionally, there should be an opportunity for piloting site-specific and region-specific measures. In fact, sites should be encouraged or required to identify their own quality improvement initiatives consistent with the mission and goals of the program, but responsive to site-specific needs assessments (which are conducted routinely as part of the reapplication process). For example, sites with teen pregnancy problems would need to focus quality improvement activities differently from sites with birth-spacing, birth-limiting, or STD problems. It is conceivable that site-specific quality improvement indicators would be drawn from the universe of indicators and piloted and validated using the AHRQ framework of indicator development (AHRQ, 2001). Validated indicators could then be shared horizontally across sites and vertically from the bottom up with program administrators for consideration as national indicators. Further, there should be opportunities to share best practices to help understand why variation exists, provide a mechanism to minimize variation, and move more sites toward benchmark goals. The overall process could work similarly to the rapid improvement cycles and learning collaboratives currently being advocated by the Institute for Healthcare Improvement (IHI, 2003).

As these measures are introduced, further research and analysis of the merits of the Title X program should be conducted using a mix of methodologies, such as provider observation or simulated patients, to document technical competence and communication skills. Additional scientific validation of the effectiveness of these interventions is needed. Whenever possible, indicator development, data collection, and reporting should be electronic, or funding for FTE support should be included so that service resources will not be used to offset this expense. Given the transient nature of the poor population, there should be a vision or capacity for shared information (regional health information network), consistent with national goals for a health information highway (NCVHS, 2000). Additionally, a widely acknowledged strength of the Title X program is IEC training tools and clinical standards. Validation of the success of the IEC modules by both providers and clients should be confirmed; replicated across all sites; shared across state and federal agencies, including STD clinics and public schools; and incorporated into medical, nursing, and residency training curricula. Broader support at the administrative and policy levels is needed to facilitate the development of a national agenda emphasizing the maternal, child, and family health benefits of family planning and preventive health services in general and the Title X program in particular. While the presumed benefits are clear, the opportunity now exists to demonstrate the quality of family planning services within the Title X program definitively and establish a causal link with reproductive health outcomes. The American public needs to learn to plan for pregnancy from puberty on, and to acquire the skills necessary to achieve personal reproductive goals.

ANNEX K-1. HEALTHY PEOPLE 2010 OBJECTIVES

Currently, there are 13 Healthy People 2010 objectives related to reproductive health that could serve as indicators of quality for the Title X program (see Annex Table K-1):

ANNEX TABLE K-1

Healthy People 2010 Reproductive Health Objectives with Targets and Baselines.

• 9.1. Intended pregnancy
• 9.2. Birth spacing
• 9.3. Contraceptive use
  • Contraceptive failure
  • Emergency contraception
  • Male involvement with pregnancy prevention
  • Adolescent pregnancy
  • Abstinence before age 15
  • Abstinence ages 15–17
  • Pregnancy prevention and STD protection
  • Pregnancy prevention education
  • Problems in becoming pregnant
  • Insurance coverage for contraceptive supplies and services

There are also 19 Healthy People 2010 objectives related to STDs (see Annex Table K-2):

• Chlamydia—Reduce infection in those aged 15–24.
• (GC)—Reduce infection.
• Syphilis—Eliminate primary and secondary syphilis from the United States.
• Herpes—Decrease percent with genital infection.
• Human papillomavirus (HPV) (developmental)—Decrease percent with HPV (can help minimize the number of high-risk subtypes associated with cervical cancer).
• Pelvic inflammatory disease—Reduce proportion of females who have ever acquired PID.
• Fertility problems—Decrease percent of women with fertility problems associated with chlamydia and PID.
• Heterosexual HIV (developmental)—Reduce HIV infections in females aged 13–24 associated with heterosexual contact.
• Congenital syphilis—Reduce congenital syphilis.
• Neonatal STD (developmental)—Reduce neonatal consequences from maternal STD.

• Responsible teen sex—Increase percent of adolescents who abstain from sex or use condoms if active.
• Responsible sex on television (developmental)—Increase number of positive messages related to responsible sexual behavior on television.
• Hepatitis B vaccine in STD clinics—Increase number of STD programs that offer hepatitis B vaccine.
• Screening in detention and jails (developmental)—Screen within 24 hours of admission and provide treatment before release.
• Contracts to treat nonplan partners (developmental)—Increase percent of local health departments that have contracts with managed care providers for treatment of nonplan partners.

• Annual screening for chlamydia (developmental)—Increase percent of women under age 25 screened annually.
• Screening of pregnant women (developmental)—Increase percent of pregnant women screened for STD, HIV, and (BV).
• Compliance with recognized STD treatment—Increase percent of primary care providers who treat patients with STDs who manage according to standards.
• Provider referral for sex partners (developmental).

ANNEX K-2. REPRESENTATIVE INDICATORS FOR EACH OF THE EIGHT BROAD CATEGORIES IN BERTRAND ET AL. (1994)

I.

Policy Environment

• Existence of a policy development plan
• Number of appropriately disseminated policy analyses
• Number of awareness-raising events targeted to leaders
• Existence of a strategic plan for expanding the national family planning program
• Integration of demographic data into development planning
• Number of statements of leaders in support of family planning
• Formal population policy addressing fertility and family planning
• National family planning coordination
• Level of the family planning program within the government administration
• Levels of import duties and other taxes
• Restrictions on advertising of contraceptives in the mass media
• Absence of unwarranted restrictions on providers and users
• Quality of program leadership
• Extent of commercial-sector participation

II.

Service Delivery Operations

• Management
• Training
• Commodities and logistics
• Information–education–communication (IEC)
• Research and evaluation

III.

Management (illustrative indicators)

• Existence of a clear mission that contributes to the achievement of program goals
• Realization of operational targets
• Clearly defined organizational structure
• Adequacy of staffing
• Awareness of current financial position
• Access to current information on key areas of program functioning
• Access to current information on program progress
• Capacity to track commodities

IV.

Training

• Number/percentage of courses that achieve learning objectives
• Number/percentage of courses that contribute to the achievement of program training objectives
• Number/percentage of courses in which the training methodology is appropriate for the transfer of skills and knowledge
• Number of trainees by type
• Number/percentage of trainees who have mastered relevant knowledge
• Number/percentage of trainees competent to provide a specific family planning service
• Number/percentage of trained providers assessed to be competent at a specified period (e.g., 6 months) post-training
• Number/percentage of trainees who apply the skills to their subsequent work

V.

Commodities and Logistics

• Pipeline wastage
• Percentage of storage capacity meeting acceptable standards
• Frequency of stock-outs
• Percentage of service delivery points (SDPs) stocked according to plan
• Percentage of key personnel trained in contraceptive logistics
• Composite indicator for commodities and logistics

VI.

Information–Education–Communication

• Number of communications produced, by type, during a reference period
• Number of communications disseminated, by type, during a reference period
• Percentage of target audience exposed to program messages, based on respondent recall
• Percentage of target audience who correctly comprehend a given message
• Number of contraceptive methods known
• Percent of audience who acquire the skill to complete a certain task as a result of exposure to a specific communication
• Percentage of target audience exposed to a specific message who report liking it
• Number/percentage of target audience who discuss message(s) with others, by type of person
• Percentage of target audience who advocate family planning practice

VII.

Research and Evaluation

• Presence of an active research and evaluation unit
• Extent of use of a service system
• Conduct of periodic household and/or special-purpose surveys and studies
• Conduct of operations research
• Regular conduct of process evaluations
• Conduct of effectiveness, efficiency, and impact evaluations
• Use of research and evaluation results for program modification
• Dissemination of research and evaluation results

VIII.

Family Planning Service Outputs

• Accessibility (illustrative indicators)
  • Number of SDPs located within a fixed distance or travel time of a given community (i.e., service density)
  • Cost of 1 month’s supply of contraceptives as a percentage of monthly wages
  • Restrictive program policies on contraceptive choice
  • Percentage of the population who know of at least one source of contraceptive services and/or supplies
  • Percentage of nonuse related to psychosocial barriers

REFERENCES

• AGI (The Alan Guttmacher Institute). 2000. Fulfilling the promise: Public policy and U.S. family planning clinics . New York: AGI.
• AHRQ (Agency for Healthcare Research and Quality). 2001. Refinement of the HCUP quality indicators. www​.qualityindicators​.ahrq.gov/data/hcup/qirefine.htm (accessed January 15, 2008).
• AHRQ. 2004. Prevention quality indicators overview . Rockville, MD: AHRQ. http://www​.qualityindicators​.ahrq.gov/pqi_overview.htm (accessed January 15, 2008).
• AHRQ. 2006. a. Inpatient quality indicators overview . Rockville, MD: AHRQ. http://www​.qualityindicators​.ahrq.gov/iqi_overview.htm (accessed January 15, 2008).
• AHRQ. 2006. b. Patient safety indicators overview . Rockville, MD: AHRQ. http://www​.qualityindicators​.ahrq.gov/psi_overview.htm (accessed January 15, 2008).
• AHRQ. 2006. c. Pediatric quality indicators overview . Rockville, MD: AHRQ. http://www​.qualityindicators​.ahrq.gov/pdi_overview.htm (accessed January 15, 2008).
• AHRQ. 2007. Screening for chlamydial infection. Rockville, MD: AHRQ. http://www​.ahrq.gov/clinic​/uspstf/uspschlm.htm (accessed May 7, 2008).
• Alden, D. L. 2004. Client satisfaction with reproductive health care quality. Social Science & Medicine 59:2219–2232. [PubMed: 15450699]
• Amaral, G., D. G. Foster, M. A. Biggs, C. B. Jasik, S. Judd, and C. D. Brindis. 2007. Public savings from the prevention of unintended pregnancy: A cost analysis of family planning services in California. Health Services Research 42(5):1960–1980. [PMC free article: PMC2254565] [PubMed: 17850528]
• Becker, D., M. A. Koenig, Y. M. Kim, K. Cardona, and F. L. Sonenstein. 2007. The quality of family planning services in the United States: Findings from a literature review. Perspectives on Sexual and Reproductive Health 39:206–215. [PubMed: 18093037]
• Bertrand, J. T., R. J. Magnani, and J. Knowles. 1994. Handbook of indicators for family planning. Chapel Hill, NC: Carolina Population Center. http://www​.cpc.unc.edu​/measure/publications/pdf/ms-94-01.pdf (accessed April 18, 2008).
• Bowman, S. A. 2006. Television-viewing characteristics of adults: Correlations to eating practices and overweight and health status. http://www​.cdc.gov/pcd​/issues/2006/apr/050139.htm (accessed May 10, 2008). [PMC free article: PMC1563980] [PubMed: 16539779]
• Bronstein, J. M., A. Vosel, S. K. George, C. Freeman, and L. A. Payne. 2007. Extending Medicaid coverage for family planning: Alabama’s first 4 years. Public Health Reports 122:190–197. [PMC free article: PMC1820443] [PubMed: 17357361]
• Bruce, J. 1990. Fundamental elements of the quality of care: A simple framework. Studies in Family Planning 21(2):61–91. [PubMed: 2191476]
• Center for Reproductive Rights. 2004. America must continue its commitment to reproductive health. http://www​.reproductiverights.org (accessed May 10, 2008).
• Cleland, J., S. Bernstein, A. Ezeh, A. Faundes, A. Glasier, and J. Innis. 2006. Family planning: The unfinished agenda. Lancet 368:1810–1827. [PubMed: 17113431]
• Collins, R. L., M. N. Elliott, S. H. Berry, D. E. Kanouse, D. Kunkel, S. B. Hunter, and A. Miu. 2004. Watching sex on television predicts adolescent initiation of sexual behavior. Pediatrics 114(3):e280–e289. [PubMed: 15342887]
• Deming, W. 1986. Out of the crisis. Cambridge, MA: MIT Center for Advanced Engineering Study.
• Donabedian, A. 1968. Promoting quality through evaluating the process of patient care. Medical Care 6:181–202.
• Dreweke, J. 2006. Each $1 invested in Title X Family Planning program saves $3.80. Doubling funding could prevent 244,000 unplanned pregnancies, result in net savings of almost $800 million . New York: Guttmacher Institute. http://www​.guttmacher​.org/media/nr/2006/11/16/index.html (accessed May 7, 2008).
• Finer, L. B., J. E. Darroch, and J. J. Frost. 2002. US agencies providing publicly funded contraceptive services in 1999. Perspectives on Sexual and Reproductive Health 34:15–24. [PubMed: 11990635]
• Fisher, E. S., and J. E. Wennberg. 2003. Health care quality, geographic variations, and the challenge of supply-sensitive care. Perspectives in Biology and Medicine 46:69–79. [PubMed: 12582271]
• Foster, D. G., C. M. Klaisle, M. Blum, M. E. Bradsberry, C. D. Brindis, and F. H. Steward. 2004. Expanded state-funded planning services: Estimating pregnancies averted by the Family PACT Program in California, 1997–1998. American Journal of Public Health 94(8):1341–1346. [PMC free article: PMC1448453] [PubMed: 15284041]
• Fowler, C. I., J. Gable, and J. Wang. 2008. Family planning annual report: 2006 national summary . Research Triangle Park, NC: RTI International.
• FPCA (Family Planning Council of America, Inc.). 1999. Performance monitoring system (draft provided by A. Stith, IOM).
• Frost, J. J. 2002. Family planning annual report: 2001 summary, part 1. New York: AGI.
• Frost, J. J. 2003. Family planning annual report: 2002 summary, part 1. New York: AGI.
• Frost, J. J., and L. Frohwirth. 2005. Family planning annual report: 2004 summary, part 1. New York: AGI.
• Gilliam, M. L., and K. Hernandez. 2007. Providing contraceptive care to low income African American teens: The experience of urban community health centers. Journal of Community Health 32:231–244. [PubMed: 17696048]
• Glasier, A., A. M. Gülmezoglu, G. P. Schmid, C. G. Moreno, and P. F. Van Look. 2006. Sexual and reproductive health: A matter of life and death. Lancet 368(9547):1595–2607. [PubMed: 17084760]
• Gregory, K. D., C. J. Hobel, L. M. Korst, M. Lu, and C. Reyes. 2005. A framework for developing Maternal Quality Care Indicators . Prepared for the California Department of Health and Human Services, Maternal and Child Health Branch.
• Guttmacher Institute. 2008. Facts in brief. Facts on contraceptive use . http://www​.guttmacher​.org/pubs/fb_contr_use.html (accessed May 8, 2008).
• HHS (U.S. Department of Health and Human Services). 2000. a. Healthy people 2010 (2nd edition). Washington, DC: U.S. Government Printing Office.
• HHS. 2000. b. With understanding and improving health and objectives for improving health (2 volumes). In Healthy people 2010 (2nd edition). Washington, DC: HHS.
• HHS. 2000. c. Healthy people 2010—Reproductive health. www​.hhs.gov/opa/pubs/hp201_rh.html (accessed April 28, 2008).
• HHS. 2001. Program guidelines for project grants for family planning services. http://www​.hhs.gov/opa​/familyplanning/toolsdocs​/2001_ofp_guidelines_complete.pdf (accessed April 24, 2008).
• HHS. 2006. Family planning program priorities, legislative mandates, key issues. http://www​.hhs.gov/opa (accessed April 24, 2008).
• IHI (Institute for Healthcare Improvement). 2003. The breakthrough series: IHI’s collaborative model for achieving breakthrough improvement . IHI Innovation Series white paper. Boston: IHI. www​.IHI.org (accessed April 28, 2008).
• IOM (Institute of Medicine). 1995. The best intentions: Unintended pregnancy and the well-being of children and families. Washington, DC: National Academy Press. [PubMed: 25121228]
• IOM. 2001. Crossing the quality chasm: A new health system for the 21st century. Washington, DC: National Academy Press. [PubMed: 25057539]
• IPPF (International Planned Parenthood Federation). 1998. Family planning programs: Improving quality (sidebars). Population Reports, Series J, No. 47.
• Jain, A., J. Bruce, and S. Kumar. 1992. Quality of services, programme efforts and fertility reduction. In Family planning programmes and fertility. Edited by J. F. Phillips, editor; and J. A. Ross, editor. . Oxford: Clarendon Press.
• Kerr, E. A., editor; , C. L. Damberg, editor; , and S. M. Asch, editor. (editors). 2000. Quality of care for women: A review of selected clinical conditions and quality indicators. Santa Monica, CA: RAND.
• Korst, L. M., K. D. Gregory, M. C. Lu, C. Reyes, C. J. Hobel, and G. F. Chavez. 2005. A framework for the development of maternal quality of care indicators. Maternal and Child Health Journal 9(3):317–341. [PubMed: 16160758]
• Kunkel, D., K. Eyal, E. Biely, et al. 2003. Sex on TV3: A biennial report to the Kaiser Family Foundation. Menlo Park, CA: The Henry J. Kaiser Foundation. www​.kff.org/entmedia/loader​.cfm?url=/commonspot​/security/getfile​.cfm&PageID=14209 (accessed April 28, 2008).
• Lindberg, L. D., J. J. Frost, C. Sten, and C. Dailard. 2006. The provision of funding and contraceptive services at publicly funded family planning agencies: 1995–2003. Perspectives on Sexual and Reproductive Health 38:37–45. [PubMed: 16554270]
• Lu, M. C., and N. Halfon. 2003. Racial and ethnic disparities in birth outcomes: A life course perspective. Maternal and Child Health Journal 7:3–18. [PubMed: 12710797]
• Meyers, D., T. Wolff, K. D. Gregory, L. Marion, V. Moyer, H. Nelson, D. Petitti, and G. Sawaya. 2008. A summary of United States Preventive Service Task Force (USPSTF) recommendations for STI screening. American Family Physician 77(6):819–824. [PubMed: 18386598]
• Mosher, W. D., G. M. Martinez, A. Chandra, J. C. Abma, and S. J. Wilson. 2004. Use of contraception and use of family planning services in the United States, 1982–2002. Hyattsville, MD: National Center for Health Statistics.
• NCQA (National Center Quality Assurance). 2007. Improving chlamydia screening. Strategies from top performing health plans. http://www​.ncqa.org/Portals​/0/Publications​/Resource%20Library​/Improving_Chlamydia_Screening_08.pdf (accessed May 7, 2008).
• NCVHS (National Committee on Vital and Health Statistics). 2000. Toward a national health information infrastructure interim report. http://ncvhs​.hhs.gov/NHII2kReport.htm (accessed April 8, 2008).
• OMB (Office of Management and Budget). 2005. The Program Assessment Rating Tool (PART). Family planning assessment. http://www​.whitehouse​.gov/omb/expectmore/detail/10003513​.2005.html (accessed April 8, 2008).
• Paine, K., M. Thorogood, and K. Wellings. 2000. The impact of the quality of family planning services on safe and effective contraceptive use: Literature review. Human Fertility 3(3):186–193. [PubMed: 11844376]
• Piotrow, P. T., editor; , D. L. Kincaid, editor; , J. G. Rimon II, editor; , and W. Rinehart, editor. (editors). 1997. Health communication: Lessons from family planning and reproductive health . Westport, CT: Praeger.
• RamaoRao, S., and R. Mohanam. 2003. The quality of family planning programs: Concepts, measurements, interventions and effections. Studies in Family Planning 34:227–248. [PubMed: 14758606]
• Roberts, D. F., U. G. Foehr, V. J. Rideout, and M. Brodie. 1999. Kids and media at the new millennium. In A Kaiser Family Foundation report: A comprehensive national analysis of children’s media use: Executive summary. Menlo Park, CA: The Henry J. Kaiser Family Foundation.
• RTI International (Research Triangle Institute International). 2006. Family planning annual report: 2005 national summary . Research Triangle Park, NC: RTI International.
• Sandfort, T. G., M. Orr, J. S. Hirsch, and J. Santelli. 2008. Long-term health correlates of timing of sexual debut: Results from a national US study. American Journal of Public Health 98(1):155–161. [PMC free article: PMC2156059] [PubMed: 18048793]
• Schunmann, C., and A. Glasier. 2006. Measuring pregnancy intention and its relationship with contraceptive use among women undergoing therapeutic abortion. Contraception 73:520–524. [PubMed: 16627038]
• Schuster, M. A., S. M. Asch, E. A. McGlynn, E. A. Kerr, A. M. Hardy, and D. S. Gifford. 1997. Development of a quality of care measurement system for children and adolescents. Archives of Pediatrics and Adolescent Medicine 151:1085–1092. [PubMed: 9369869]
• Sofaer, S., and K. Firminger. 2005. Patient perceptions of the quality of health services. Annual Review of Public Health 26:513–559. [PubMed: 15760300]
• Speroff, T., and G. T. O’Connor. 2004. Study designs for PDSA quality improvement research. Quality Management in Health Care 13:17–32. [PubMed: 14976904]
• Sullivan, T. M., and J. T. Bertrand. 2000. Monitoring quality of care in family planning by the quick investigation of quality (QIQ): County reports. MEASURE Evaluation Technical Report Series No. 5. Chapel Hill, NC: Carolina Population Center.
• Thornburn, S., and L. M. Bogart. 2005. a. African American women and family planning services: Perceptions of discrimination. Women’s Health 42:23–39. [PubMed: 16418120]
• Thornburn, S., and L. M. Bogart. 2005. b. Conspiracy beliefs about birth control: Barriers to pregnancy prevention among African Americans of reproductive age. Health Education and Behavior 32(4):474–487. [PubMed: 16009745]
• United Nations. 1994. International Conference on Population and Development (ICPD), September 1994, Cairo, Egypt. http://www​.un.org/popin/icpd2.htm (accessed April 8, 2008).
• Wennberg, J. E. 1999. Understanding geographic variations in health care delivery. New England Journal of Medicine 340:52–53. [PubMed: 9878647]
• WHO (World Health Organization). 1998. WHO definition of quality. Population Reports Series J, No. 47.
• WHO. 2004. Medical eligibility criteria for contraceptive use (3rd Edition, Reproductive Health and Research). Geneva, Switzerland: WHO.
• Winter, L., and A. S. Goldy. 1987. Staffing patterns in family planning clinics: Which model is best? Family Planning Perspectives 19(3):102–106. [PubMed: 3622738]
• Zaky, H. H., H. A. Khattab, and D. Galal. 2007. Assessing the quality of reproductive health services in Egypt via exit interviews. Maternal and Child Health Journal 11:301–306. [PubMed: 17237993]