NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.
Committee on the Outcome and Impact Evaluation of Global HIV/AIDS Programs Implemented Under the Lantos-Hyde Act of 2008; Board on Global Health; Board on Children, Youth, and Families; Institute of Medicine. Evaluation of PEPFAR. Washington (DC): National Academies Press (US); 2013 Jun 27.
OVERVIEW
The committee's evaluation employed a mix of methods and layers of investigation and analysis involving a range of primary and secondary data sources, taking into account the methodological design considerations described in Chapter 2. This included mapping of investments using financial data, assessing trends over time using program monitoring indicators and clinical data from the Office of the U.S. Global AIDS Coordinator (OGAC) and PEPFAR implementing partners, benchmarking progress against stated programmatic targets and goals, reviewing extensive documents, and analyzing primary data collected through more than 400 semi-structured interviews with a range of stakeholders on visits to 13 PEPFAR partner countries, at the U.S. headquarters (HQ) of PEPFAR, and at other institutions and multilateral agencies.
Primary and secondary data were analyzed, using appropriate methodologies, by the members of the evaluation committee, the Institute of Medicine (IOM) study staff, and consultants with specialized knowledge in both qualitative and quantitative methodologies. The contracted consultant for quantitative methodologies was a biostatistical firm in Washington, DC, Statistics Collaborative, Inc. (SCI), and for qualitative methodologies was Dr. Sharon Knight in Greenville, North Carolina. The committee, IOM staff, and consultants took steps to assess and ensure the quality and completeness of the data used for the evaluation, and took these factors into account during data interpretation. The methods used to ensure the quality of the primary data collected and the secondary data received through data requests are described in more detail in the sections that follow. When externally analyzed data were used, the committee, IOM staff, and consultants reviewed and assessed the quality of the data and the methodologies used.
As described in Chapter 2, the mandate of the committee was to draw conclusions and make recommendations across the whole of the PEPFAR initiative. Wherever possible, data were gathered and data analyses and interpretation were conducted and presented across all 31 of the PEPFAR partner countries that were the focus of the evaluation;1 however, only very limited data were comparable and comprehensive across all countries. In order to not limit the committee's findings to data consistently available across the whole of the program and all of these countries, which would have been a significant constraint, the evaluation drew on those subsets of countries, programmatic areas, or intervention components implemented within PEPFAR for which sufficient data could be gathered to contribute to the assessment. Therefore, data presentations and analyses representing these subsets were interpreted with care to inform conclusions about the whole of the program. For example, analysis of country visit interview data was limited to the countries selected for visits by the committee. In addition, some analyses drew on existing data sources that were available only for some countries, programs, and partners, such as Track 1.0 partner data. Some evaluation questions were most applicable only for a subset of countries, such as countries with concentrated epidemics driven by injecting drug use. Finally, the time and resources available limited the scope of some analyses, such as those involving review of Country Operational Plans (COPs), for which the sheer volume of the documents over all countries and years limited the feasibility of comprehensive review across all countries. Throughout the report, where data analyses that do not represent the whole of the program are presented, the scope of these data is described. Because the committee was not charged to draw conclusions or make recommendations at the level of specific countries, partners, or programs, analyses of data from subsets of countries or partners are presented in a manner designed to maintain anonymity.
By applying this mix of methods and layers of investigation and analysis using a range of available primary and secondary data sources, the committee arrived at findings that could be triangulated to draw conclusions about the performance and impact of PEPFAR, even when there was no one data source that was sufficient or one methodological approach that was feasible. Building on the interpretation of the available data, the conclusions and recommendations presented in this report represent the consensus reached through the deliberations of the evaluation committee. Over the course of the evaluation, the full committee met six times in person, with participation of the IOM staff and consultants. One additional meeting was conducted using Web-based conferencing. In addition, working groups within the committee that were focused on specific content areas held additional meetings by teleconference as needed for ongoing deliberations as well as for data analysis and interpretation. These committee activities were augmented by ongoing communications via telephone and e-mail among the committee members, staff, and consultants.
The following sections describe some of the overarching processes that the committee used to frame and shape the evaluation. Subsequent, more detailed sections describe the methods for each of the data sources used in the evaluation.
Development of Evaluation Questions and Mapping of Data Sources
Through working groups consisting of a subset of committee members, the evaluation committee identified proposed evaluation questions based on major content areas, the statement of task (see Appendix A), the Program Impact Pathway (PIP) framework (see Chapter 2), and the preliminary work reported in the Strategic Approach (IOM and NRC, 2010). Once the working groups established their initial questions and subquestions, IOM staff and consultants developed and provided to the committee the following information pertaining to each of the questions:
- The domains of the PIP to which the question belonged (i.e., input, activity, output, outcome, or impact)
- The type of data necessary to answer the question (e.g., financial data; program monitoring, surveillance, and clinical data; qualitative interview data; literature and document review)
- A description of potential data sources that had been identified
- Limitations associated with the data sources, such as issues related to availability, the feasibility of accessing the data, and any other relevant issues that could inform considerations for formulating data requests and for the utility of the data
Mapping of Potential Data Sources
The IOM staff and consultants then carried out an extensive data-mapping effort for more than 150 evaluation questions, building on the preliminary work conducted during the strategic planning and operational planning phases. The data-mapping process relied on document review, stakeholder interviews, information obtained from preliminary data requests, and information gathered during 2 pilot country visits. The data mapping served to assess the feasibility of collecting and using data from each source, taking into consideration the burden that data requests would place on each source's resources and staff time. In addition, this data mapping assessed whether data from each source would require new data analysis in order to answer the evaluation questions posed by the committee.
The categories of available data sources that were mapped and ultimately used for the evaluation included financial data; program monitoring, surveillance, and clinical data; qualitative interview data; and literature and document review. The sources included central OGAC data, data from multilateral organizations, data from implementing partners, and data from publicly available documents and other sources. The data sources used for the evaluation are described in more detail in subsequent sections of this appendix.
Priority Evaluation Questions
Committee members then worked with IOM staff and consultants to finalize a set of priority evaluation questions based on relevance to the statement of task and related evaluation considerations, relative importance among subquestions, and feasibility of answering each question with the time, resources, and data available. The ultimate relative contribution of data sources to different content areas and evaluation questions and, ultimately, to the committee's conclusions and recommendations varied depending on data availability and appropriateness.
Overview of Data Collection
A summary of the data request and data collection processes for each major data source is provided in the sections that follow, along with a description of the analyses for which the data were used.
Requests for interviews and requests for secondary data not readily publicly available were made by the IOM independently, with OGAC and partner country mission teams serving as a liaison only when necessary. Participation in the evaluation was voluntary. Except when reference is made to existing published materials, findings, examples, and comments are not attributed to individuals, and the identities of individuals, programs, partners, and countries are protected.
FINANCIAL DATA
Global Financial Data
To contextualize PEPFAR's financial contribution within the broader donor funding landscape for HIV/AIDS, the committee examined disbursement data on official development assistance for HIV/AIDS as reported to the Organisation for Economic Co-operation and Development (OECD) Creditor Reporting System (OECD, 2012). Disbursements represent the sum of two OECD sector codes: sexually transmitted disease (STD) control (which includes HIV/AIDS) and the social mitigation of HIV/AIDS. The committee examined data for the 31 PEPFAR countries that were writing COPs when the IOM evaluation study process began in 2009.
PEPFAR Financial Data: Available, Obligated, and Outlaid
Each quarter, OGAC submits summary financial status reports to Congress on “the allocation, obligation and expenditure of funds appropriated for [PEPFAR]” (OGAC, n.d.-b, p. 1). These reports are publicly available. The committee used the fourth-quarter report from each fiscal year (FY) from 2004 through 2011 to calculate annual appropriations, obligations, and outlays for the PEPFAR program (OGAC, n.d.-b).
PEPFAR Financial Data: Annual Expenditure Data Calculated from Agency Reporting
In May 2012, in response to a committee data request, SCI received from OGAC PEPFAR funding obligations and outlays for FYs 2004 through 2011 for all countries receiving PEPFAR funding. Upon review of the data and through clarifications with OGAC, IOM staff and consultants realized that these financial data corresponded to the cumulative amount of funding available, obligated, and outlaid from each budget year rather than the actual annual amount of funding available, obligated, and outlaid. Another request was made to OGAC for funding data that would clearly distinguish funding by budget year and reporting year and that would represent actual annual expenditures, regardless of the year in which the money was appropriated or obligated. In July and August 2012, SCI received from OGAC cumulative agency-specific funding for each reporting year. Annual expenditures were derived as described below.
Data Description
OGAC sent 78 Excel spreadsheets containing financial data for the 6 agencies that received PEPFAR funding between FY 2004 and FY 2011:
- Department of Defense (DOD)
- Department of Health and Human Services (HHS)
- Department of Labor (DOL)
- Department of State (STATE)
- Peace Corps (PC)
- U.S. Agency for International Development (USAID)
With the exception of STATE, each agency reported all of its financial information to OGAC in a consolidated format. STATE, however, reported its PEPFAR funding through five distinct offices/bureaus:
- Bureau of African Affairs (AF)
- Bureau of East Asian and Pacific Affairs (EAP)
- Bureau of Population, Refugees and Migration (PRM)
- Bureau of Western Hemisphere Affairs (WHA)
- Office of the U.S. Global AIDS Coordinator (OGAC)
Each file contained cumulative budget information on available, obligated, and outlaid funds, by country, for each FY. As discussed in Chapter 4, most PEPFAR funding does not have an annual use-or-lose requirement (i.e., unspent funding from one FYcan often be carried over to be spent in subsequent years) (OGAC, 2008). Therefore, the money spent during a particular year had the potential to come from the budgets of multiple prior FYs.
Consolidating Data into Consistent Files
Each agency provided funding to a different group of PEPFAR countries, many of which received funding from more than one agency. Therefore, each agency's set of budgetary files was first consolidated into a single data file, creating a total of 10 unique datasets—one set per agency or bureau. Each country's annual funding was retained within each data file to enable potential analyses that would require subgroups of PEPFAR countries based on country attributes. Next these datasets were harmonized into a single dataset to allow for data to be used together to comprehensively represent PEPFAR spending across agencies, in total and by country.
Documenting Discrepancies, Notes, and Comments
The data extraction process revealed embedded comments within spreadsheet cells and footnotes explaining data nuances; this information was recorded in a separate file. Additionally, some funding numbers changed from one reporting year to the next. Increases in funding amounts were expected over time as more of the funding from a particular fiscal year was expected to be obligated or outlaid. Decreases, however, were not expected from one year to the next, i.e., the amount of available funding from a specific fiscal year budget was not expected to decrease in subsequent reporting years. Therefore, these unexpected changes in the funding data were documented.
For all three of these situations—embedded comments, footnotes, or unexpected changes in funding—the following information was recorded for each instance:
- Agency/bureau—which agency's spreadsheets contained the comment, footnote, or inconsistency
- Country/region—the country or region affected by the comment, footnote, or inconsistency
- Reporting year—the reporting year with the observation
- Budget year—the year during which the budget was issued
- Comment, footnote, or inconsistency—verbatim comments and footnotes from the spreadsheet; inconsistencies were described as clearly as possible
- Detected by agency or IOM—an indicator variable reflecting whether the comment or footnote was already in the spreadsheet or whether the inconsistency was encountered by IOM staff during the data extraction process
To further assess the most notable discrepancies in the available totals by country and by year, these were compiled in a separate spreadsheet and compared from the inception year through 2011. In particular, major discrepancies occurred when the dollar amounts reported as available for a given budget year changed (both increases and decreases were observed) in subsequent reporting years, although one would expect the amount to be a fixed constant for a budget year after that year in which it was made available. These discrepancies ranged in magnitude with a maximum difference of $214 million between two reports for one budget year for one agency. As a result, it was difficult to assess which were the correct figures for the total amounts made available. Overall, the number of discrepancies and magnitude of changes from year to year diminished in later reporting years, and the same degree of discrepancy was not seen in the reporting of outlays.
Calculation of Annual Expenditures
Once the funding data were completely extracted into a single data file, serial subtractions of each reporting year's cumulative outlay data were performed in order to obtain the amount of money actually spent (outlaid) during each reporting year, regardless of the fiscal year's budget from which the money came. To get the annual expenditure for a given FY, all prior year outlays were subtracted from the cumulative total outlays reported for that year. Given the data discrepancies described above, in calculating the annual expenditures, the data for all of the FYs were taken from the FY 2011 reports in order to have one consistent source that reflected the most recently available data.
Quality Control
When all of the data had been extracted into consistent data files, SCI compared all the extracted data files against the raw data files sent from OGAC. The validator worked with the original data extractor and reconciled all inconsistencies uncovered within the extracted data files. This independent validator also verified the serial equations used to calculate the amount of funding spent during each reporting year. The validated datasets were not reconfirmed with OGAC.
Data Presentation
Once all of the data had been validated, SCI imported the data into the analytic software SAS® version 9.3,2 which it used to generate financial presentations of the annual expenditure over time. These presentations were provided to the committee in November 2012.
PEPFAR Financial Data—Planned/Approved Funding for All PEPFAR Countries
Planned/approved funding reflects how OGAC and PEPFAR mission teams plan to obligate and outlay funds. Each year, OGAC releases an operational plan for PEPFAR that includes summary budget information regarding the planned and approved use of PEPFAR funding, including which activities will be implemented by which agencies, as determined during the interagency planning process. These PEPFAR Operational Plans report planned/approved funding for four technical areas that correspond to the primary categories of HIV/AIDS services and systems strengthening efforts: Prevention, Care, Treatment, and Other. PEPFAR funding is planned through budget codes which capture funding information about more specific activities within these categories (OGAC, 2011c).
Data Extraction Process
The planned/approved funding was extracted from the PEPFAR Operational Plans by year and by budget code (OGAC, 2005a, 2006c, 2007b, 2008, 2010, 2011b,c). Planned/approved funding data were extracted independently by two IOM staff into identical spreadsheets; records were then compared across datasets to identify and correct inconsistent values. Each staff member extracted data on total PEPFAR funding by implementing agency and year, as well as total PEPFAR funding by budget code and year.
Data Presentations
The data extraction was validated and SCI converted the data into constant 2010 USD to allow for a consistent interpretation of funding over time. The consultants then generated a final dataset to be used for data presentations showing funding by agency, type of program, and budget code.
PEPFAR Financial Data—Planned/Approved Funding for Subset of 31 Countries
Data on planned/approved funding from the subset of 31 countries that were the focus of this evaluation were gathered through a separate data extraction. These data were used for the committee's analysis of funding by country characteristic.
Data Extraction Process
The planned/approved funding was extracted according to the following classifications:
- By country (31 countries in total)
- By year (FY 2005 through FY 2011, for the years during which a given country was completing a COP)
- This funding information was extracted from the following publicly available data sources, which were determined to be the most comprehensive across the classifications for the data extraction (OGAC, n.d.-a,c):
- ο
FY 2005 through FY 2007—Focus countries only: PEPFAR Operational Plans
- ο
FY 2008—Focus countries: PEPFAR Operational Plans; non-focus countries: individual COPs
- ο
FY 2009—All countries: individual COPs
- ο
FY 2010 and FY 2011—All countries: PEPFAR Operational Plans
SCI developed specifications corresponding to the variables necessary for the PEPFAR financial data extraction process and developed dataset specifications for two separate extraction processes. The first data extraction compiled annual, country-specific funding by agency; the second data extraction compiled annual, country-specific funding by technical area and budget code. Data were not extracted by both agency and technical area, but rather either by agency or by technical area. During the extraction process, any funding corresponding to regions (e.g., Central America, Central Asia, and Caribbean) was omitted, and the process was limited to the 31 countries that were preparing COPs at the time this evaluation was initiated. Funding amounts were rounded to the nearest whole dollar.
During the extraction by technical area, some budget codes switched from one technical area to another across reporting years; however, these differences were tracked in an effort to make consistent comparisons over time.
Data Extraction Quality Control
Two SCI consultants extracted the data independently into comparably formatted spreadsheets. Each consultant extracted a spreadsheet of funding data by year, country, and agency, as well as a second spreadsheet by year, country, technical area, and budget code. Once all of the data had been extracted across all budget years, one of the consultants developed a tool to compare individual records across datasets and to flag inconsistent values. This comparison tool flagged every instance of a record with inconsistent information, whether it corresponded to how the extractors recorded a particular budget code or whether the budget amounts differed. Together, the consultants then reconciled the inconsistent records. Once their datasets matched 100 percent, a third, independent SCI consultant imported the data into SAS software and used a random number generator to select 50 of the 1,302 records (about 4 percent) that summarized the financial information by agency and 80 of the 4,123 records (about 2 percent) that summarized the financial information by budget code; the consultant then crosschecked these 130 values against the information written in the PEPFAR Operational Plans and COPs. Some of these records corresponded to countries, agencies, or budget codes that were not specified during a particular year. Therefore, this selection of records also confirmed that particular combinations of years, countries, agencies, and budget codes were not inadvertently incorporated into the datasets. All 130 validation records matched the operational plans exactly, thus confirming the quality of the data extraction process.
Data Presentations
Once the validation process was complete, SCI generated a final dataset to be used, along with publicly available data from global sources, for data presentations showing PEPFAR funding by HIV prevalence, average funding per person living with HIV, and country income level.
Planned/Approved Funding by Prime Partner
The committee examined planned/approved funding data extracted from a range of publicly available data sources. The process of extracting and compiling these data was time intensive, so to be feasible within the resources and time available for the study, the committee's analysis had to be limited to a subset of partner countries. The committee chose to compile these data for the same 13 countries purposefully selected for country visits, as described later in this appendix. Within this subset of countries, the committee was able to compare partner data and planned/approved PEPFAR funding for the focus countries for FY 2004 through FY 2010 and for non-focus countries for FY 2008 through FY 2010.
Data Extraction
Data were extracted according to the following characteristics:
- By country
- By year
- By prime partner
This funding information was extracted from the following data sources, which were determined to be the most comprehensive data sources available across the classifications for the data extraction. For FY 2004 to FY 2006, the prime partner funding data were extracted from a Center for Global Development (CGD) dataset in order to present the most complete data consistently for those years (CGD, 2008). The CGD publicly released this dataset which “was originally obtained from the State Department by the Center for Public Integrity through several Freedom of Information Act requests and a lawsuit against the U.S. Government, settled out of court” (Oomman et al., 2008, p. 8). This dataset contains PEPFAR data on country funding obligated to prime partners in focus countries in FYs 2004, 2005, and 2006. Before releasing the dataset, CGD added data on central funding that is obligated from OGAC headquarters to partners to implement programs in countries. For FY 2004 and FY 2005, CGD obtained central funding information from the Center for Public Integrity; FY 2006 funding was estimated by CGD based on previous funding amounts and the total allocation of PEPFAR funding for focus countries in FY 2006 (Oomman et al., 2008). Partner lists that provide the amount of funding obligated to prime partners within a country are also available on PEPFAR's website. For focus countries, these lists are available with funding information for FY 2005 to FY 2008; for non-focus countries, these lists are available only for FY 2008 (OGAC, n.d.-d). By comparing most of the PEPFAR partner lists and the CGD dataset for FY 2005 and FY 2006, it appears that the PEPFAR partner lists for 2005 include country and central funding, but the PEPFAR partner lists for 2006 include only country funding. Therefore, the CGD dataset was determined to be the most complete data for FY 2004 to FY 2006 and it allowed the presentation of both country and central funding consistently for these years.
For FY 2007 and FY 2008, data were extracted from PEPFAR partner lists. PEPFAR partner lists provide funding amounts but do not include information about the type of funding (i.e., central or country funding) (OGAC, n.d.-d). There is no dataset that is equivalent to the CGD dataset for FY 2007 and FY 2008, so the committee was unable to determine whether the FY 2007 and FY 2008 partner lists report country and central funding or only country funding.
The only sources of partner data for FY 2009 and FY 2010 are the COPs; these data are limited to planned partner funding (not obligations). Since not all partners have been chosen by the time the COPs are submitted, these are incomplete sources of partner funding. For example, 2 percent of total funding data extracted from the FY 2009 COPs was labeled as “To Be Determined,” which means that a partner had not yet been chosen or contracted with to provide planned activities (OGAC, n.d.-a).
Two IOM staff members extracted the data independently into identical spreadsheets; records were then compared across datasets to identify and correct inconsistent values. After the data extraction, staff carried out additional research as needed to determine the type (multilateral, government, nonprofit, for-profit, or academia) and origin (U.S.-based, partner country–based, multilateral, or other) of each prime partner. The type and origin of each partner was also recorded in the same spreadsheets with the funding information.
Data Extraction Quality Control
After all the data and supplementary information had been extracted, one of the IOM staff developed a tool to compare individual records across spreadsheets and to flag inconsistencies. This comparison tool flagged every instance of a record with inconsistent information. Independently, the staff went back to the original sources to confirm or edit the inconsistencies and the spreadsheets were compared again. Any inconsistencies that persisted were reconciled together until all records matched.
Data Limitations
The prime partner data compiled by IOM staff for the committee were limited by the incompleteness of the data sources, as described above. Overall, the total amount of partner funding compiled for this analysis reflects only 77 percent of the total planned/approved funding for this subset of countries for FY 2005 to FY 2010 (as reported in the operational plans). The gap between the data used for the analysis and the total planned funding represents expenses not expended through the COP prime partner mechanism, To Be Determined (TBD) funding, and any central funding not reported in partner lists or COPs. Funding may have been reported as TBD if prime partners had not been identified prior to reporting or in situations where partners had been identified, but contracts had not been finalized. Given the nature of these gaps, the funding not represented in the dataset would be distributed across types of partners, therefore the committee determined that the dataset represented adequate information for a reasonable descriptive analysis to help understand the distribution of prime partner funding.
Data Presentations
Once all of the data were validated, a final Excel dataset was used to produce data presentations showing funding over time by type and origin of prime partner using data aggregated across all 13 countries.
PEPFAR PROGRAMMATIC INDICATOR DATA
Background
PEPFAR indicators are used to monitor and assess progress in the HIV/AIDS response within and across PEPFAR-funded technical areas (the collection and use of PEPFAR indicators are discussed in depth in Chapter 11, “Knowledge Management”). Indicators can grouped into two overall categories: those that were collected by partner countries during FY 2004–FY 2009 and those that are being collected starting in FY 2010, after a revision of the indicators (OGAC, 2009b). Prior to FY 2010, there were two levels of PEPFAR indicator collection and reporting requirements. Indicators were either required and reported to OGAC or were recommended but not reported to OGAC (OGAC, 2005b, 2007a). The first round of indicator guidance, released in 2005, defined 65 indicators to be reported annually to OGAC (OGAC, 2005b, 2007a).3 The next indicator guidance, issued in 2007, increased the number of centrally reported indicators to 76 (OGAC, 2005b, 2007a). Seven of these indicators corresponded to overall country-level indicators (see Table C-1). These seven country-level indicators were composed of two components, direct and indirect, where direct results represented “counts of uniquely identified individuals receiving prevention, care, and/or treatment services at a unique program or delivery service point that receives USG funding,” while indirect results represented “contributions made by the USG to overall systems strengthening and capacity building that occur apart from, and at higher levels than the actual points of service delivery,” resulting in PEPFAR contributions to the national program results (OGAC, 2007a, p. 12).
TABLE C-1
Country-Level Indicators Reported During FY 2004–FY 2009.
In 2009, OGAC developed the Next Generation Indicators (NGIs) to reduce the number of PEPFAR-specific reporting requirements and, where possible, to align with globally harmonized and reported indicators in partner countries. As a result, starting in FY 2010 PEPFAR classified indicators according to three reporting levels: essential reported to OGAC HQ, essential not reported to OGAC HQ, and recommended (OGAC, 2009b).4 The NGIs “reflect PEPFAR's strategy to increase country owner ship of HIV/AIDS efforts and ensure that host countries are at the center of decision-making, leadership, and management of their HIV/AIDS programs” (OGAC, 2009b, p. 6). As a result, the number of “essential reported” indicators decreased, and countries are now required to report only 25 programmatic indicators to OGAC. If a partner country has a signed Partnership Framework, the country is also required to report a 26th programmatic indicator to OGAC (H6.1.D). Five additional indicators are routinely reported to OGAC, but these are national-level indicators and are not PEPFAR-specific. In addition, other revisions to indicators were made, including removal of indicators, addition of new indicators, and efforts to make definitions easier to understand and information easier to collect (OGAC, 2009b).
The NGIs include eight primary indicators, a subset of the 25 programmatic indicators required to be reported to OGAC (see Table C-2) (OGAC, 2011a). These are akin to the prior country-level indicators shown in Table C-1. Table C-3 shows how the overlapping country-level indicators before and after the NGI revision map to one another.
TABLE C-2
Primary Indicators for PEPFAR Next Generation Indicators (FY 2010–Present).
TABLE C-3
Overlapping Country-Level Phase 1 and Primary Phase 2 Indicators.
Because some of the indicators reported to OGAC changed over time, IOM staff and consultants performed a data mapping of the indicators. Staff and consultants grouped the indicators into three distinct categories (FY 2004 through FY 2007, FY 2008 through FY 2009, and from FY 2010 onward); compared the indicator definitions from each indicator guidance document; linked indicators that had essentially the same definition across indicator guidance; and recorded whether each indicator was essential reported, essential not reported, or recommended during each of the three time periods. After this mapping was completed, the committee was able to determine which of the indicators had been reported to OGAC each year as well as how many years of data OGAC had obtained for each indicator.
In addition to mapping the alignment over time, the PEPFAR indicators were also mapped to the 25 UNGASS (United Nations General Assembly Special Session) indicators. One notable difference between the two sets of indicators is that UNGASS indicators are reported by calendar year, whereas PEPFAR reports in fiscal years (October through September). Furthermore, UNGASS indicators reflect country-level results, whereas only a select few PEPFAR indicators report on country-level outcomes (UNAIDS, 2009). These differences limited the feasibility of using the annual reported data for these indicators together in analyses or data presentations.
Requests for Data
In response to a series of formal data requests to OGAC between April 2011 and March 2012, IOM staff and consultants received a subset of the PEPFAR indicator data that had been reported to OGAC from FY 2004 through FY 2010. In April 2011, IOM staff and SCI first conducted a phone interview with Dr. Paul Bouey, the Deputy U.S. Global AIDS Coordinator responsible for Strategic Information and Budget and Management, to obtain a preliminary assessment of available PEPFAR indicator data. IOM conducted a follow-up, in-person interview with Dr. Bouey and two other OGAC staff members later that month to learn more about OGAC data collection and querying processes. After the phone interview and again during the in-person interview, IOM staff and consultants formally requested from OGAC all centrally reported PEPFAR program monitoring data corresponding to FY 2004 through FY 2010. Data requests were not made for PEPFAR indicators that had not already been reported centrally (“essential not reported” or “recommended” indicators) because these data would not be available consistently across all PEPFAR countries and because, based on the time and burden that would be placed on country programs, it was determined that it would not be feasible to collect these data for the evaluation.
At the time of the initial data request, OGAC indicated that it would make only the core programmatic indicators described in the previous section (seven from FY 2004–FY 2009 and eight from FY 2010) available to IOM. Additionally, OGAC indicated that the 17 remaining NGI programmatic indicators (FY 2010) were undergoing internal data querying and that OGAC would not be able to share them until October 2011. As a result, the first official data request to OGAC included the seven core FY 2004–FY 2009 programmatic indicators and eight available core FY 2010 programmatic indicators, which OGAC provided. In September 2011, SCI requested the remaining 17 NGI (FY 2010) programmatic indicators, which OGAC provided in November 2011 following OGAC's internal querying and cleaning processes.
Between this first and second request, SCI sent OGAC a series of questions containing data clarifications related to the first set of indicators received. These questions dealt primarily with the interpretation of country-specific direct and indirect indicators and how they related to the indicator targets that OGAC had defined.
After committee members worked with IOM staff and consultants to finalize a set of priority evaluation questions (PEQs) based on relevance to the statement of task and related evaluation considerations (as described earlier), IOM staff and consultants selected a subset of the centrally reported FY 2004–FY 2009 program monitoring data deemed most relevant and useful to answer the PEQs and evaluate PEPFAR. This subset of Phase 1 data was prioritized into three tiers (see list below) to space out the burden of data requests on OGAC. SCI made the initial three-tiered request to OGAC in November 2011, with a clarification of the request provided in January 2012. At the end of January 2012, SCI also made a fourth request for national-level indicators not explicitly included in previous requests. The resulting four tiers were
- Tier 1 – FY 2004–FY 2009 indicators considered most broadly useful in answering the priority evaluation questions, therefore these indicators were requested as early as possible.
- Tier 2 – FY 2004–FY 2009 indicators with linked NGIs and presumed more likely to be queried and available.
- Tier 3 – FY 2004–FY 2009 indicators that do not have corresponding NGIs but are as important for evaluation as the other tiered indicators. These indicators were made a lower-priority request in case these indicators required more time to prepare than the other listed tiers of indicators.
- Tier 4 – NGI (FY 2010) national-level indicators, which had not been included with the initial transfer of NGI programmatic indicators.
Although OGAC readily provided program monitoring data for the core targets and indicators, it did not provide several of the requested FY 2004–FY 2009 indicators because it considered the data too unclean to be useful. Furthermore, several of the FY 2004–FY 2009 indicators that had been reported to OGAC were disaggregated by sex, age, or other pertinent characteristics. IOM requested disaggregated data for all indicators for which it was collected. OGAC did not provide disaggregation for any of the FY 2004–FY 2009 indicators the IOM evaluation team requested, but it did provide disaggregated information for all NGI indicators.
Data Presentations
Once SCI received the PEPFAR indicator data from OGAC, it created tabular and graphical presentations of the data over time using SAS software versions 9.2 and 9.3. Data presentations were provided to the committee in June 2011, September 2011, April 2012, and June 2012.
Data Limitations
One of the main limitations of the centrally reported PEPFAR indicator data was a lack of suitable denominators, which made it difficult to assess the coverage achieved both with respect to the population being served at PEPFAR sites or programs and the total population in need at the national level. The latter would have been useful to determine PEPFAR's contribution to national coverage. In many cases, the most suitable denominators available were the targets set forth by OGAC for each country during each reporting year. OGAC described the target-setting process as “complex” (Bouey and De Leon, 2011). Initially countries were assigned 5-year targets based on 50 percent of the country's estimated need. This was later transitioned to a process whereby countries determined their own targets, a process that OGAC described as developing more useful and more realistic targets (Bouey and De Leon, 2011). OGAC provided the annual targets for each indicator for each country between FY 2004 and FY 2010. PEPFAR target setting is discussed in more depth in Chapter 11.
Additional limitations for specific indicators are presented in the chapters where the data are presented.
TRACK 1.0 PARTNER DATA
When PEPFAR was initiated, some funding for programs was centrally managed through the Centers for Disease Control and Prevention (CDC) and the Health Resources and Services Administration (HRSA) via what are known as “Track 1.0” awards. These were one-time, 5-year awards given to organizations with existing operations in focus countries and a proven track record, and therefore with the capacity to respond and implement programs quickly. The intent of this approach was to rapidly initiate and scale up prevention, care, and treatment services in PEPFAR focus countries. These partners reported to both in-country Mission Teams and to CDC and HRSA directly (McCullough and Miller, 2009; Sessions, 2006). As a result of this implementation design, Track 1.0 partners have been collecting data longer than other implementing partners; as will be described further in the sections that follow, these partners also often collect data beyond the indicators that OGAC requires for routine reporting.
Four PEPFAR Track 1.0 partners have been involved in the early and ongoing implementation of care and treatment programs, including antiretroviral therapy (ART)5:
- AIDSRelief, a consortium of five organizations:
- ο
Catholic Relief Services, as prime grantee
- ο
The Institute of Human Virology at the University of Maryland School of Medicine, as technical lead for clinical care and treatment
- ο
Futures Group, as lead agency for strategic information
- ο
Catholic Medical Mission Board, as an implementing partner
- ο
IMA World Health, an implementing partner
- Columbia University's International Center for AIDS Care & Treatment Programs
- Elizabeth Glaser Pediatric AIDS Foundation
- Harvard University
Beginning in March 2011, IOM staff and SCI contacted CDC headquarters staff involved in the central management of Track 1.0 partners as well as each Track 1.0 partner to initiate a discussion about their roles within PEPFAR and to engage them in discussions related to potential sharing of data. IOM staff and consultants also requested lists of indicators collected, corresponding data dictionaries, and associated data collection guidance from each partner. In response to requests for more information, SCI received from each Track 1.0 partner materials pertaining to their work through PEPFAR.
In July 2011, SCI followed up with a more formal query that elicited additional information from each Track 1.0 partner, including a list of the programs they established within each of their partner countries; implementation of public health evaluations; challenges and barriers in the field; intended data usage; data quality assessments; employee training; data management, both within and across partner countries; and standardized data reporting across all Track 1.0 partners.
Between September 2011 and February 2012, IOM staff and consultants conducted semi-structured, in-person interviews with each Track 1.0 partner. One purpose of these interviews, which were part of the primary interview data collection process for the evaluation, was to learn more about the data that each Track 1.0 partner had collected and analyzed, as well as to explore the possibility of acquiring data and analyses from the partners.
CDC Track 1.0 Data
IOM participated in a teleconference with CDC staff in early March 2012 to discuss the possibility of sharing the data from all Track 1.0 partners. Each Track 1.0 partner is required to compile data from quarterly, facility-based reporting forms and report the information to the CDC and HRSA (CDC manages these data for the partners whose grants are administered through both CDC and HRSA). IOM requested that it receive these quarterly care and treatment data directly from CDC. CDC indicated it would share a de-identified dataset that did not identify implementing Track 1.0 partners, countries, or facilities. At the time of the request, data were available from FY 2005 through the end of FY 2011.
Data Summary
In April 2012, SCI received from CDC an Excel spreadsheet containing 7 years of quarterly, facility-level data (Q1 FY 2005 through Q4 FY 2011) from all four Track 1.0 partners, representing programs in thirteen PEPFAR partner countries. The individual records were not identified by facility, Track 1.0 partner, or country. This dataset contained more than 20,000 records, each of which corresponds to a specific facility's data from a single quarter between FY 2005 and FY 2011. During some quarters, as many as 1,300 facilities reported information to the CDC. Each facility reported up to 200 different values. The data related to patient care, treatment, and adherence; median CD4 counts at baseline and at 6 and 12 months after treatment initiation; training in ART and HIV care; and patient retention, death, and loss to follow up. Other variables, such as specific ART regimens, had been captured during the earlier years of this program; however, these data were ultimately removed from the quarterly form and were therefore not included in the dataset provided by CDC.
In addition to containing several years of data, this dataset also included disaggregated data. All of the data corresponding to patient enrollment into clinical care, initiation of ART treatment, and patient follow-up are disaggregated by sex (male/female) and age (0–14 years and 15+ years). From FY 2008 onward, the pediatric data are disaggregated by narrower age ranges: 0–1 years, 2–4 years, and 5–14 years; these narrower age ranges are also disaggregated by sex. The training information is disaggregated by type of worker (physician, nurse, or other). Track 1.0 partners also reported the total number (no disaggregation) of people trained in HIV palliative care (now referred to as Care and Support services) within each country.
Relationship of Track 1.0 Data to OGAC Indicators
This Track 1.0 dataset contained three variables that are reported to OGAC and that ultimately contribute to the following PEPFAR programmatic indicators (OGAC, 2009b):
- C2.1.D – Number of HIV-positive adults and children receiving a minimum of one clinical care service (Care and Support Sub Area, Clinical Indicator)
- T1.1.D – Number of adults and children with advanced HIV infection newly enrolled on ART (Treatment Indicator)
- T1.2.D – Number of adults and children with advanced HIV infection receiving antiretroviral therapy (CURRENT) (Treatment Indicator)
The Track 1.0 partners reported each of these indicators with sex and age disaggregation. The data in the CDC dataset corresponded to fiscal quarters, whereas OGAC indicators represent FYs.
Data Limitations
CDC de-identified the data provide to SCI. Each facility was assigned an arbitrary facility ID number that prevented linking the information to a specific facility name. Further, instead of indicating within which country a facility is located, CDC assigned each of the Track 1.0 countries with an arbitrary number between 1 and 13 and removed any partner-specific identifier from the data. This de-identification made it impossible to con duct focused analyses. Without partner, country, and facility identifiers, the data could not be linked to partner-specific or country-specific information that had been acquired through other data sources, such as structured interviews.
Data Quality
Until this data request, CDC had reviewed the data only in aggregated forms not at the facility level; the CDC did not conduct preliminary data checks before providing the dataset to SCI. Instead, to increase efficiency, CDC worked closely with SCI to address issues and inconsistencies that arose during data quality checks conducted after the transfer of data; identified issues were in the facility-level dataset and would not have affected prior aggregated data use by CDC.
During a preliminary review of the dataset, SCI discovered various quality issues that were eventually reconciled. For example, some facilities were missing records for some quarters, while other facilities had multiple records for other quarters. Upon closer review, CDC realized that their “matching” program, which matches and links records from the same sites over time, had erroneously matched multiple sites to one another. Considering the sheer magnitude of data—during peak Track 1.0 partner involvement, CDC received quarterly data from as many as 1,300 sites which were reported in a variety of languages—a few inconsistencies like this were expected. The CDC corrected these mismatches and sent SCI a revised dataset in May 2012.
The dataset also contained other occasional erroneous information, which appears to have been attributable to data entry errors. For example, the sum of the men and women receiving treatment within a given facility did not always add up to the total number of people receiving treatment at that facility. Such inconsistencies were infrequent, however, and the magnitude of the difference was usually small. Another example involved CD4 data: Some records indicated that the number of people whose CD4 counts were included in the calculation of median CD4 count was equal to the median CD4 count (e.g., 15 people were in the cohort, and the reported median CD4 count equaled 15). Records for which these types of errors were identified were omitted from any analyses conducted by IOM staff and consultants.
Analyses
SCI prepared data presentations for the committee pertaining to changes over time in enrollment into clinical care services and treatment services, active facilities, and persons trained. The disaggregated nature of the care and treatment data allowed for assessments of potential differences between age groups (e.g., adults versus children, infants versus older children) and between males and females. Data presentations were provided to the committee in June 2012.
Individual Track 1.0 Partner Data
As described previously, during semi-structured interviews, teleconferences, and e-mail communication between March 2011 and October 2012, IOM communicated with all of the Track 1.0 partners individually to learn more about their programs and discuss the feasibility of their sharing data and analyses with IOM.
One partner was willing to share its existing analyses and to conduct some limited additional analyses for the evaluation related to survival, patient retention, health systems, treatment access and coverage, baseline CD4, monitoring for treatment failure, quality of service delivery, program management and capacity building. There was not adequate time to provide data for SCI to conduct independent analyses, which would have required a time-consuming internal approval process, nor for this partner to conduct extensive new analyses for the evaluation. The partner provided analyses using aggregated data to SCI over several transfers between May and October 2012. Because the scope of this mandated study does not include a country- or partner-specific evaluation, prior to sharing these analyses with the committee and before including data presentations in the report, SCI redacted country names and partner affiliations from all country-specific analyses. The Track 1.0 partner worked closely with IOM staff and consultants to resolve any questions or issues with regard to the analyses.
A second Track 1.0 partner was willing to explore the possibility of sharing existing analyses with IOM and possibly conducting new analyses; however, upon closer consideration, the partner declined to share any analyses because it determined that sharing these analyses for use and publication in this evaluation might interfere with its ability to subsequently publish results in peer-reviewed journals. A third Track 1.0 partner also declined to share data or analyses. The fourth Track 1.0 partner had limited independently collected data beyond the quarterly data reported to CDC and OGAC, and therefore did not have extensive additional data or analyses to consider sharing for this evaluation.
Track 1.0 Partner Publication Data
To augment the Track 1.0 partner data received, SCI made a comprehensive request to the partners to identify and collect published results based on their data that could potentially contribute to the committee's findings. Three of the four Track 1.0 partners shared comprehensive publication lists.
GLOBAL DATA SOURCES
In addition to pursuing PEPFAR-specific data through the centrally-reported indicators and the Track 1.0 partner data, IOM staff and consultants reviewed additional potential data sources for global HIV/AIDS data. As the committee finalized its PEQs for each content area, the committee work groups, staff, and consultants mapped data from these global data sources to the PEQs to determine which data to prioritize. The mapping effort focused on such data sources as UNAIDS, Demographic and Health Surveys (DHS), the World Bank, the International Epidemiological Database to Evaluate AIDS (IeDEA), WHO Global Health Observatory, and the United Nations Children's Fund (UNICEF). Information gathered in the data mapping process led to collecting the data from those sources that were deemed to be most relevant, appropriate, available, and feasible to use within the scope, time, and resources of the study. The data ultimately used in the evaluation are described where data are presented in the report chapters. Additional data sources considered for use in impact analyses considered by the committee are described in the sections below.
Data Sources for Overall Data Mapping
AIDSinfo, UNAIDS Database (UNAIDS, 2011)
The UNAIDS Database is an interactive system that allows the end user to query a compilation of national and international data sources including the World Health Organization (WHO), UNICEF, UNAIDS, and Measure DHS on the topics of demography, development, epidemiology, HIV/TB, law, and spending. Data covering the following topics are available for at least 169 countries:
- UNGASS indicators
- HIV and AIDS prevalence estimates
- Data pertaining to orphans
- Country population data
- Development measures such as life expectancy, infant mortality rate, etc.
International Census Data (U.S. Department of Commerce, n.d.)
This is a searchable database maintained by the U.S. Census Bureau, with population estimates and projections based on census, survey, vital statistics, and other data available by country or area. Data are available for countries and areas with current populations of 5,000 or more beginning in 1950 with projections up to the year 2050. Available estimates and projections include
- Birth, death, and growth rates, migration rates, infant mortality, and life expectancy
- Fertility rates
- Total population and population by age and sex
WHO Global Health Atlas (WHO, n.d.-a,c)
The WHO Communicable Disease Atlas database contains reports, documents, and data on some of the major infectious diseases of poverty, although at the time of the data mapping for this evaluation it had no specific data on HIV. The WHO Global Atlas of the Health Workforce contains global data corresponding to the health care workforce, including community health workers and laboratory health workers.
WHO Global Health Observatory (WHO, n.d.-b)
The Global Health Observatory is a statistics repository for the WHO. An end user can export data by country, over time (from 1990), and on various health topics, including
- Health-related Millennium Development Goals (poverty and hunger, child mortality, maternal health, HIV/AIDS, malaria, environmental sustainability, and global partnerships for development)
- Mortality and burden of disease by country (life expectancy, morbidity and mortality, disability-adjusted life-years, disease, and injury)
- World health statistics (mortality and burden of disease, cause-specific morbidity and mortality, selected infectious diseases, health services coverage, risk factors, health workforce and infrastructure, health expenditures, health inequities, and demographic and socioeconomic statistics)
- Immunization (country and regional data)
- Nutrition (child malnutrition)
- Epidemic-prone diseases (cholera)
- Tobacco control
- Violence and injury (road safety)
- Global Health Information Systems on Alcohol and Health (production and availability, levels and patterns of consumption, harms and consequences, economic aspects, alcohol control policies, prevention, research, and treatment)
- HIV/AIDS (data on the size of the epidemic and on the HIV/AIDS response)
- Tuberculosis (cases, diagnosis, drug regimes, and treatment success)
- Public health and environment (household and air pollution, outdoor air pollution, water, sanitation and hygiene, lead, second-hand smoke, UV radiation, climate change, occupational risk factors, total environment, and children's environmental health)
International epidemiologic Databases to Evaluate AIDS (International epidemiologic Databases to Evaluate AIDS, n.d.)
IeDEA is an initiative that establishes regional centers to collect and harmonize data across countries. The centers collect key variables to address research questions in HIV/AIDS that cannot currently be answered by single cohorts. IeDEA provides a mechanism to pool data being collected around the world to enhance HIV/AIDS research.
One data center has been funded for each of the seven IeDEA regions (North America, the Caribbean and Central and South America, Asia and Australia, West Africa, Central Africa, East Africa, Southern Africa). This program includes data from nearly 525,000 HIV-infected persons from 43 different countries.
World Bank (World Bank, n.d.)
The data catalog of the World Bank and the World Data Bank provide access to indicators from the World Bank datasets which cover a wide variety of topics and countries. The datasets contain times series data which can be downloaded by country (all data for all years for a single country), by topic (specific indicators for all countries and years), or by individual indicator (all countries for all years). This group of databases covers a variety of topics (economic, health, financial, etc.). Some of the data goes as far back as 1960. Databases include, but are not limited, to
- World Development Indicators and Global Development Finance
- Health Nutrition and Population Statistics
- Africa Development Indicators
- Education Statistics
UNICEF (UNICEF, 2011)
UNICEF provides statistics tables that show economic and social statistics with a particular focus on child well-being. End users can choose countries and indicators to view particular data or to download the data into spreadsheets. Data cover the following topics:
- Basic indicators (total population and annual numbers of deaths and births)
- Adolescents (population, marital status, attitudes toward domestic violence, education, and HIV knowledge)
- Child protection (child labor, child marriage, and attitudes toward domestic violence)
- Demographics (birth rate, crude death rate, and population under 18 and under 5)
- Economics (gross domestic production [GDP] per capita and inflation rates)
- Education (literacy rates, phone use, and enrollment ratios)
- Health (percent receiving Expanded Program on Immunization vaccinations and individual vaccination rates)
- HIV/AIDS (prevalence in adults, pregnant women, and young adults, condom use, orphaned children, and HIV knowledge)
- Nutrition
- Rate of progress (under-5 mortality rates, fertility rate, and GDP annual growth)
- Women (literacy, maternal mortality, and antenatal care)
DHS HIV/AIDS Survey Indicators Database and STATcompiler (USAID, 2011)
Measure DHS is a compilation of data from sample surveys. The indicators included are derived primarily from the UNAIDS National AIDS Programmes: Guide to Monitoring and Evaluation. This database goes back as far as 1985, although the earliest date varies greatly across countries. The main sources of HIV/AIDS indicators in this database are
- AIDS Indicator Survey (AIS)
- Behavioral Surveillance Surveys (BSS)
- Demographic and Health Surveys (DHS)
- Multiple Indicator Cluster Surveys (MICS)
- Reproductive Health Surveys (RHS)
- Sexual Behavior Surveys (SBS)
STATcompiler includes indicators from the RHS. Datasets and reports for these surveys include information related to
- Household characteristics
- Fertility
- Family planning
- Other proximate determinants of fertility
- Fertility preferences
- Early childhood mortality
- Maternal and child health
- Maternal and child nutrition
- HIV/AIDS
- Malaria
DOLPHN (USAID, n.d.)
The Data Online for Population, Health, and Nutrition (DOLPHN) database pulls select demographic and health indicator data for various countries directly from multiple data sources, including BUCEN, CDC, Census Bureau, DHS, UNAIDS, United Nations Educational, Scientific and Cultural Organization, UNICEF, the United Nations Development Programme, WHO, and the World Bank. This data repository is a website that allows quick and easy access to several indicators from different data sources at one time.
Exploration of Adult Mortality Data for Impact Analyses
As described in Chapter 6, during its deliberations, the committee explored the possibility of conducting new modeling to evaluate the impact of PEPFAR. Ideally the committee would have liked to design a model to determine if a larger annual investment of PEPFAR funding, as a continuous variable over time, had led to a greater impact on health.
The committee focused on a possible model of impact on adult all-cause mortality with potential outcome and explanatory variables, using the country as the unit of analysis. Such an approach would require a simple dataset with time-varying data for country-specific mortality, a measure of PEPFAR expenditure as the independent variable, and a judiciously chosen set of relevant covariates. Some additional variables were considered as baseline covariates (e.g., total country population) rather than time-varying factors to prevent oversaturating the models.
The committee opted not to try to model HIV mortality since cause-specific mortality data are limited in many low- and middle-income countries and, even when they are collected, they may not include many HIV-related deaths that have a different specific proximal cause (e.g., an opportunistic infection) (IOM and NRC, 2010). The committee also discussed modeling child mortality as a function of PEPFAR funding over time but decided against it due to the small number of countries where HIV is a sizeable contributor to childhood mortality (see Chapter 7 for a more in-depth discussion of child mortality in PEPFAR partner countries).
Mortality Data Sources and Their Limitations
IOM staff and SCI reviewed potential sources of mortality data that could be used in longitudinal analyses that would be designed to understand the overall impact of the PEPFAR program. Each of the sources considered is described briefly here. Vital or civil registration systems are not adequate in many low- and middle-income countries, and where systems exist, they are often not comprehensive and there is underreporting or misreporting of cause of death (IOM and NRC, 2010). Several sources produce estimates of mortality using modeling methods; the most common limitation among available estimated mortality data was that explanatory variables planned to be included in this modeling, such as HIV treatment coverage, were used in the calculation of the mortality estimates, thereby creating circularity in the potential modeling.
UNAIDS Spectrum/EPP model (UNAIDS, n.d.) This model has been used to produce annual, country-specific estimates of the number of AIDS deaths starting in 1990. Spectrum/EPP modeling requires the end user to input country-level data pertaining to HIV prevalence and ART coverage. Using treatment coverage and HIV prevalence as explanatory variables in our analyses would thus have resulted in circularity within the model.
WHO Statistical Information System (WHOSIS) (WHO, n.d.-d) WHOSIS provides estimates of the total number of registered deaths by country and year, beginning with 1979, including cause of death, sex, and age. The data available are compiled from national vital registration systems, with underlying cause of death as recorded and reported by the relevant national authority. The site also includes information on estimates of coverage (of all estimated deaths) and estimated completeness of the registered deaths by country (proportion of all deaths that are registered in the population covered by the vital registration system for a country). Although the site has annual mortality estimates for many countries, several PEPFAR countries (e.g., Angola and Botswana) are not represented in the dataset.
U.S. Census Bureau (U.S. Department of Commerce, n.d.) This database contains annual crude death rates from 1950 and projected out through 2050 for foreign countries (crude death rate is the average annual number of deaths during a year per 1,000 people at mid-year). The mortality estimates in this database include the impact of ART for selected countries. The number of adults and children receiving or targeted to receive ART comes from OGAC, WHO, and other sources. ART coverage is projected by assuming a constant yearly percent reduction in the unmet need for ART, with the assumption that 80 percent is considered universal coverage (U.S. Department of Commerce, 2010).
Central Intelligence Agency (CIA) World Factbook (Central Intelligence Agency, n.d.) The CIA World Factbook provides annual death rates per 1,000 people (all ages) for all countries for the years 2000 through 2011. Several requests were made to the identified contact person on the website to clarify the sources of these mortality estimates, but no response was received. Without knowing more about the data sources and how these estimates were calculated, it was not possible to determine whether circularity would arise with the proposed longitudinal modeling.
United Nations Population Division (United Nations, n.d.) This site contains estimates of adult mortality rates between ages 15 and 60 years, for both sexes separately and combined. It also provides estimates of the crude death rate (deaths per 1,000 people). Both sets of estimates are provided for 5-year periods rather than the annual estimates needed for the modeling that was considered.
World Bank (World Bank, n.d.) The World Bank reports crude death rates by country and year. The primary reference listed for these data is the UN Population Division.
Institute for Health Metrics and Evaluation (IHME) (Institute for Health Metrics and Evaluation, n.d.) This site provides global estimates of adult mortality risk (probability of death between the ages of 15 and 60 years) between 1970 and 2010. Unlike the data on the United Nations Population Division's website, however, the IHME rates are provided annually. Data are disaggregated by sex. For these data a database was compiled of “3,889 measurements of adult mortality for 187 countries from 1970 to 2010 using vital registration data and census and survey data for deaths in the household corrected for completeness, and sibling history data from surveys corrected for survival bias” and a Gaussian process regression was used “to generate yearly estimates of the probability of death between the ages of 15 years and 60 years for men and women for every country with uncertainty intervals that indicate sampling and non-sampling error” (Rajaratnam et al., 2010, p. 1).
Mortality estimates from Bendavid et al. Through phone and in-person meetings in June and July 2012, IOM staff and SCI spoke with Dr. Eran Bendavid, the primary author of two publications evaluating the PEPFAR program, to learn more about the methods used to develop mortality estimates (Bendavid and Bhattacharya, 2009; Bendavid et al., 2012). The mortality rates used in the 2012 published modeling were derived from raw DHS data (Bendavid et al., 2012). The smoothing of year-to-year variation that results from modeling was eliminated by calculating mortality estimates from these raw data (although smoothing can also be an advantage because it prevents irregularities in data from creating false impressions of change). The calculated mortality rates also avoided the circularity introduced when potential explanatory variables are incorporated in the mortality estimation. These data were therefore considered promising as a viable option for longitudinal analyses which was newly available as of the 2012 publication. However, whereas the committee was interested in modeling using annual by-country mortality estimates, Bendavid's dataset included more than 9 million data points corresponding to individual mortality outcomes, and were only calculated for a subset of PEPFAR countries. Although these data could in theory be converted into annual by-country mortality estimates and calculated for additional countries, this process would have required more time than was available before completion of the evaluation.
PEPFAR Financial Data Sources and Their Limitations
Ideally, the model considered by the committee would have used the annual PEPFAR investment by country for all countries as the independent variable. This would have allowed for the use of a continuous variable to represent the magnitude of PEPFAR funding in place of the dichotomous variable of focus versus non-focus countries, which was a limitation in prior published longitudinal modeling (see discussion in Chapter 6). However, as described in Chapter 4, PEPFAR funding data is typically reported as cumulative spending by the budget year in which funding was made available rather than the actual amount expended each year. In addition, information on PEPFAR funding disaggregated by partner countries is not publicly available. As described previously, after several iterative funding data requests, SCI received from OGAC cumulative country-specific and agency-specific funding reports from which annual expenditures by country could be manually derived. However, these data represented only a subset of total PEPFAR funding, and concerns about completeness and accuracy in the data limit their utility for longitudinal modeling.
Decision to Omit Longitudinal Modeling from the Evaluation
After careful consideration, the evaluation committee determined that, given the limitations of the available mortality and financial data, within the scope, time, and resources of this evaluation, it was not feasible to conduct statistical analyses comparing countries with variable levels of PEPFAR funding over time to correlate changes in key outcome or impact indicators. Ultimately, the limitations were determined to be too great to design and carry out analyses that would meaningfully add to the previously described existing analyses in the published literature (see discussion in Chapter 6).
DOCUMENT REVIEW
The evaluation team conducted targeted and systematic document reviews and also appraised purposefully selected documents in an effort to gather facts, particularly those of relevance to countries involved in the evaluation. As part of the data collection process associated with the evaluation, the team's desk review included reviewing several types of documents: those providing PEPFAR-specific process, policy, and planning guidance documents; PEPFAR operational plans; reports from PEPFAR-supported activities and evaluations; global guidance documents related to HIV; country-specific HIV/AIDS reports; reports from multilateral agencies and other organizations external to PEPFAR; and the peer-reviewed literature.
The specific documents reviewed are referenced where the information gathered is used throughout the report. Generally speaking, the review of documents was advantageous in gathering credible, accessible information, some of which was not available through other evaluation methods. Examples of the types of key documents that the committee reviewed include guidance from the WHO on HIV/AIDs testing, diagnosis, treatment, care, support, and prevention and key country-specific reports such as the biannual Country HIV/AIDS Progress Reports to the UN Secretariat (UN General Assembly Special Session reports). The team also reviewed PEPFAR annual reports to Congress, PEPFAR COPs, and multiple years of COP guidance from OGAC to mission teams. The review of multiple years of COPs for a number of countries provided annual country-specific work plans that were developed collectively by USG agencies under the leadership of the U.S. ambassador for the purpose of determining annual goals, resource plans, planned activities, and implementing partners (OGAC, n.d.-a). Also included in the process and planning guidance were Partnership Frameworks and their Implementation Plans, which provided insight into the processes, steps, and measures to support country ownership and the transition to country-led management of sustainable HIV responses (OGAC, 2009a). The evaluation team also reviewed the peer-reviewed literature related to HIV/AIDS program research, some of which was PEPFAR-sponsored, directly related to PEPFAR, or related to specific evaluation topics in countries of interest.
INTERVIEW DATA
Overview
The evaluation team of IOM committee members and staff, working with the consultation of Dr. Sharon Knight, implemented its qualitative evaluation processes systematically in an effort to ensure that the outcome of its cross-country data collection, analysis, and findings was both high-quality and consistent. The team used multiple strategies to ensure the credibility of the qualitative component of the evaluation. These strategies included triangulation, purposeful and snowball sampling, prolonged engagement in the field, attaining data saturation, researcher reflexivity, maintenance of an audit trail, participant validation of data summaries, debriefing and synthesis processes, and ensuring the accuracy of the data collected. This section describes how these strategies were applied pragmatically in the evaluation. The process of collecting the country visit interview data is summarized at the end of this section in Figure C-1.
Just as triangulation among different yet complementary data sources was an important methodological approach for the mixed-methods evaluation overall, the incorporation of triangulation within the qualitative evaluation component provided an opportunity for the research team to explore and gain insight into the PEPFAR program using diverse data sources and types as well as multiple investigators. Triangulation is an often-recommended strategy for incorporating different elements and viewpoints and can encompass triangulation among data sources, methods, and investigators (Merriam, 2002, 2009; Patton, 2002) (Rossman and Rallis, 2012). The evaluation team used these types of triangulation as part of the interview data collection and analysis process. First, data triangulation was reflected in the involvement of different sources of data through the inclusion of interviews from a range of countries and stakeholders; these interviews are described in more detail below. The use of multiple data collection strategies, such as in-depth interviews and site visit observations as well as review of documents and quantitative data relevant to each country visit, is another example of the use of methodological triangulation. Finally, investigator triangulation reflected the presence of multiple researchers in the evaluation process, including committee members, IOM staff, and consultants.
![FIGURE C-1. Country visit qualitative data collection process.](/books/NBK207010/bin/p775.jpg)
FIGURE C-1Country visit qualitative data collection process
Collection of Country Visit Interview Data
Country visit selection From November 2010 to February 2012, the evaluation committee, IOM staff, and consultants conducted 13 country visits (including 2 pilot visits) to collect interview data. Originally, 14 countries, including the 2 pilot countries, were selected as country visit sites. These countries were selected through purposeful sampling based on committee deliberations during a closed committee meeting. According to Patton (2002), purposeful sampling is a technique that specifically targets and uses the selected sample to gain insights and learn from subjects that have perspectives for the topic or phenomenon of interest and not for generalization to a population. To make country selections, committee members reviewed background data compiled by IOM staff for each of the 31 PEPFAR countries that were the focus of the evaluation. Background data covered a range of variables, including but not limited to country income level, geographical location, HIV epidemic type, HIV prevalence, status as a PEPFAR focus country, population size, PEPFAR funding per capita and per person living with HIV, and the relative contribution of PEPFAR to the national response compared to the Global Fund. Committee members eliminated for consideration those countries where the safety of the delegation team would likely be compromised based on information from the U.S. Department of State (DoS) and other trustworthy sources of information for international travel. They then iteratively grouped countries by different variables and ultimately selected a sample of countries representing a cross-section of attributes. In consultation with Mission Team staff, two originally selected countries were not visited because of security warnings of sufficient magnitude to warrant U.S. diplomatic mission travel restrictions. One additional country with similar characteristics within the original selection criteria was added to minimize the effects of the two cancellations on the selected sample.
Interviewee selection process During each country visit, the evaluation team conducted qualitative interviews with key stakeholders involved in the HIV/AIDS response. During the months leading up to a country visit and during the visit itself, the team used purposeful sampling for the initial selection of in-country interviewees. To develop a sample of interviewees that represented a range in types and levels of key stakeholders involved in the country's HIV/AIDS response, the selection process took into account input from committee members, country background research completed by the IOM staff team, targeted focus areas within the overall evaluation plan, input from the USG PEPFAR Mission Team, country visit timing, and interviewee availability. In deference to cultural norms and with respectful intentions, the team typically initiated the sampling process in a partner country by contacting key leaders associated with the PEPFAR country program and the partner-country HIV response. Sources for such information included the PEPFAR country coordinator or an individual serving in a comparable role and leaders in the partner countries' health sector. In these discussions, the IOM study co-director who was the Team Leader for the country visit, elicited information about the mission team structure, roles, and operation for PEPFAR implementation and, where possible, lists of all implementing partners by program area were used by the IOM staff to conduct additional research, which was followed by the identification and selection of interview and site visit candidates. Country-specific documents available to the team (e.g., COPs, national strategic plans for HIV/AIDS, and many others) served as additional resources to identify service organizations, programs, and individuals relevant to the response within specific countries. IOM staff then initiated contact with these resources in an effort to further identify individuals with direct experience related to various elements of the HIV response in a given country and to schedule interviews and site visits for each country visit, except where protocol required communication and scheduling with partner country government officials through formal communications by the Mission Teams. In these limited cases, documentation was sent from the IOM to the Mission Teams to present to officials. In countries where English was not the primary language, all information sent ahead about the study and requests for interviews and site visits by the IOM study co-directors were translated by language professionals hired by the IOM team, and, when needed, professional interpreters were hired to accompany the evaluation team on interviews and site visits.
Once the evaluation visit teams were in-country, the team incorporated an additional process of snowball sampling during interviews with stakeholders to identify other information-rich individuals associated with the HIV response in the country for consideration either to fill interview or site visit slots that were intentionally left available for such interviews, or else to serve as replacements in case originally scheduled interviews were cancelled by the interviewees due to conflicts in scheduling that arose after the interview or site visit had been confirmed. One example of such sampling efforts was the query, “Can you share with us the names of individuals, programs, or services that you think might contribute interesting or valuable information (sometimes on a particular issues to replace previously scheduled interviewees) regarding PEPFAR or the HIV response in (name of country)?” Another typical query was, “We are interested in learning more about successes and challenges related to the HIV response in (name of country). Are there others involved in (a particular focus of the evaluation—e.g., HIV prevention efforts) who you think we should talk to or who might be helpful to us?” An evaluation team member then contacted each potential interviewee to ascertain her or his availability during the time frame of the planned country visit and willingness to be interviewed and to schedule a possible interview time and date. Unless these were replacement interviewees, these interviews were then added to the number that had been scheduled before arriving in the country.
Although the interviews conducted in each country were tailored to the particular country and its unique HIV-response strengths, challenges and attributes, the evaluation team aimed to systematically interview stakeholders serving in particular roles in every country visited. The roles of these individuals or groups of individuals included
- U.S. country mission leader(s)
- PEPFAR all-staff Mission Team members
- PEPFAR country coordinator or equivalent
- PEPFAR technical staff/work group members (with varying numbers of staff and kinds of workgroups, depending on the country)
- Partner country stakeholders including, at minimum, health- and HIV-related government or government agency personnel, partner country HIV-related nongovernmental organizations (NGOs) directors, staff, volunteers, and others.
- U.S.-based stakeholders, including, at minimum, HIV-related NGO directors/chiefs of party, staff, or volunteers, and others
- HIV-related civil society organization leaders or members, including faith-based programs, human rights programs, and other organizations that provided programs and services for populations at elevated risk or other vulnerable populations
- Personnel having direct experience with a particular focus for data collection within a given country (e.g., services for populations at elevated risk, services for orphans and vulnerable children, the health care system, and prevention of mother-to-child transmission). Such personnel might be from different levels of a partner country government, an NGO, or U.S.-based NGOs or other organizations.
Interviewees were selected based on who had the most direct experience with the area of focus and who was willing to be interviewed. For example, interviewees who directed or provided services or programs that addressed the needs of HIV-related vulnerable children were sought in countries designated as target countries for such information.
Based on an IOM Institutional Review Board review and exemption, individual beneficiaries were not included in the interview sample, except when individuals serving in one of the roles described above were also beneficiaries of PEPFAR-supported programs (for example, some peer educators were also patients or clients of PEPFAR-supported programs).
During the process of sampling and consequent data collection, team members assiduously protected individual's confidentiality and anonymity. At no time did the evaluation team share with others external to the team the identity of anyone contacted by team members, scheduled for an interview, or interviewed. Team members provided no feedback to individuals who proffered the names of potential interviewees and, with the exception of drivers and interpreters hired by the team, divulged no information of any kind to anyone external to the team. Thus, the team kept the identity of interviewees confidential during all country visits.
Over the 13 country visits, the IOM delegations conducted a total of 383 interviews; 68 of these included a visit to a service delivery facility or program site. The interviewees included individuals or groups representing partner country governments; USG mission staff, including the DoS and the PEPFAR implementing agencies; multilateral organizations; NGOs; academia; and the private sector. Table C-4 summarizes the number of total interviews completed by stakeholder type and subtype.
TABLE C-4
Country Visit Interviews by Stakeholder Type.
In-country data collection As advocated for the conduct of credible qualitative studies (Creswall, 2007; Merriam, 2002; Patton, 2002), this evaluation involved the investigators in extended time in the field, which enabled in-depth data collection opportunities that extended well beyond a “snapshot view” (Rossman and Rallis, 2012, p. 65) of the PEPFAR program. Prolonged engagement afforded the evaluation team an opportunity to gain an in-depth understanding of the PEPFAR program in the context of each country visited.
Country visits typically spanned 2 weeks, and qualitative data collection involved an average of 25 in-depth interviews with key stakeholders as well as several site visits in each of the selected countries. Each committee member actively participated in data collection and preliminary analysis processes for at least one visit to a partner country, with the majority engaging in 1-week data collection and analysis efforts while some participated in a full 2 weeks, and a few participated in two country visits. At least one and sometimes both IOM study directors led the collection of data and data analysis processes during each country visit. Other IOM staff members also participated as delegation team members in collecting data and engaging in preliminary analysis during what was typically a full 2-week visit. Consultants participated in the data collection and analysis process for either 1 or 2 weeks during visits to seven countries.
IOM staff and consultants developed data collection toolkits that were provided for use by the evaluation teams on country visits. Additional preparation was provided through staff trainings, an initial briefing on methods and processes at a committee meeting, and an in-country orientation briefing at the beginning of each country visit. For each country visit delegation, briefing materials were prepared that included background information on the country, the national HIV response, and the PEPFAR country program, as well as basic financial, program monitoring, and surveillance data.
A subset of evaluation team members was present and engaged in qualitative data collection typically for a period of 12–14 days during each visit to a partner country, with a total in-country visit time across the selected countries of about 180 days. Thus, the team had a total of approximately 6 months of residence (or approximately 140 person-weeks for all of the committee, IOM staff, and committee consultants) in a total of 13 partner countries from the initiation of on-site data collection in November 2010 to its completion in February 2012. Each day in-country involved the evaluation team in some aspect of the evaluation process, including logistical planning for collecting data, as well as data collection, transcription of notes, team debriefings, and data analysis and interpretation.
Through semi-structured interviews, delegation team members learned about the national HIV/AIDS response, interviewees' experiences with PEPFAR, and the role of PEPFAR in the national response currently and over time. These interviews were conducted using interview guides tailored for each interview. The development of each guide was informed by the interviewee role and level, the agency type, and the program area. Guides were developed by selecting and tailoring a subset of interview questions and follow-up prompts from a pre-established set of key country visit interview questions. The development of these pre-established questions was based on the Strategic Approach to the Evaluation of Programs Implemented Under the Tom Lantos and Henry J. Hyde U.S. Global Leadership Against HIV/AIDS, Tuberculosis, and Malaria Reauthorization Act of 2008 (IOM and NRC, 2010) and the evaluation committee's priority evaluation questions. The questions covered seven primary evaluation areas: Knowledge Management, Resource Flow, PEPFAR Implementation, Programmatic Context, PEPFAR Effects, Health System Strengthening, and PEPFAR Transition to Sustainability and Country Ownership, as well as opening and final questions. In addition to the questions, follow-up prompts were developed both within these evaluation areas and also for the following programmatic areas: Prevention, Treatment, Care and Support, Laboratory, Children and Adolescents, and Gender.
Interviews were conducted by an interview team that was generally composed of a lead facilitator, a co-facilitator, and a note taker. The typical duration of an interview was 60 to 90 minutes. During the interview, the facilitator's role was to build rapport and facilitate the interview using the semi-structured guide, and the co-facilitator's role was to ask follow-up or clarifying questions, serve as time keeper, write notes, and provide an end-of-interview review of the main points heard during the interview. The purpose of the co-facilitator's summary of main points was to provide an opportunity for participant validation, which is discussed below. Additionally, the review of main points served as a starting point for the peer debriefings described below. The note taker's role was to capture and record the data by means of handwritten or typed field notes as well as electronically, via an audio recorder, with interviewee consent. Additional team members who were present during an interview served as additional note takers; if fewer team members were present, one individual served as both note taker and either facilitator or co-facilitator.
The evaluation delegation team participated in 68 field visits to a variety of sites in the 13 partner countries visited. The purpose of the site visits was to gain a contextual understanding of PEPFAR programs and resources. Each site visit included an interview, some of which were in-depth, open-ended interviews, while others were informal interviews conducted during walking tours of the sites. At least one designated delegate team member visiting the site took handwritten field notes during the visit; these were reviewed and reconciled by team members using the same procedures as interview notes, which is discussed below.
The team conducted the majority of interviews in English. In the case of interviewees who preferred conducting the interview in a language other than English, the delegation team hired professional interpreters from the partner country and oriented them to the purpose and process of qualitative data collection and to their role in the process.
Participant validation of data summaries A commitment to anonymity and confidentiality and a focus on cross-country data reporting precluded the sharing of country-specific findings with interviewees and their agencies or organizations during or after individual partner country visits. Interviewees were able to assess the scope and content of their key messages, however, in response to an end-of-interview summary of key messages that the co-facilitator offered at the conclusion of every interview. Following the summary of key points or messages, co-facilitators explicitly invited interviewees to convey any additions, corrections, or additional information that they wished to offer. Thus, all interviewees had an opportunity to affirm, modify, or extend their key messages, a process that not only affirmed that their viewpoints had been clearly understood and documented by the interview team, but also verified the accuracy and completeness of key messages shared with the team.
Researcher reflexivity Because delegation team members served as “instruments” of qualitative data collection, they were aware of the need to be reflexive and have a “simultaneous awareness of self and other and of the interplay between the two” (Rossman and Rallis, 2012, p. 10). In other words, engagement in reflexivity facilitated individuals' emergent self-awareness of personal predilections, assumptions, biases, and beliefs so that each individual could potentially recognize and thus minimize her or his impact on interviewees and the research environment as well as the impact of the research environment on them (Patton, 2002). Team leaders and consultants urged all team members to engage in reflection and reflexivity throughout the evaluation by using at least one of two primary strategies: maintaining a private reflective and reflexivity journal or engaging in verbal reflexivity during any of the interview or team debriefings. Members of the evaluation team frequently, openly, and voluntarily shared their self-awareness of personal assumptions, biases, and beliefs verbally during one or more of the multiple peer debriefings and synthesis processes associated with data collection. At times, peers encouraged a team member to be reflexive when that individual's personal assumptions or biases emerged during discussions and debriefings related to the evaluation. During discussions, it was not unusual for team members to reference a personal need for self-reflexivity regarding some topic. Thus, the need for all investigators to become increasingly self-aware about their personal beliefs, assumptions, values, and biases that could impact the research or the research environment and vice versa was frequently reinforced during each country visit.
Audit trail Maintaining an audit trail served as a means for the evaluation team to establish study credibility and confirmability (Wolf, 2003). Evaluation team members were charged with organizing and maintaining various electronic and hardcopy audit trail evidence related to the evaluation. Evidentiary documents related to the process of the evaluation included
- An agenda log maintained electronically for each country visit chronicled interview scheduling and contact information, evaluation-related contacts, and information on the participants and questions covered in completed interviews.
- An activity log maintained electronically throughout the evaluation process chronicled process and methodological decisions and action items both within and across country visits.
- Analysis and interpretation notations were indicated on flip chart paper and electronic notes during facilitated team debriefings and the mid-week and exit synthesis process. When evaluation team members recounted interviewees' viewpoints and experiences related to evaluation topics, they not only reported the content of interviewees' perspectives (“what they said”) gleaned during interviews, but also differentiated interviewees' narratives from how they as team members interpreted what interviewees shared with them. Team members also discussed emerging linkages among participants' interview data and other data such as documents and observations.
- A codebook was initially developed and then revised based on evaluation topics and, to a lesser degree, data that emerged from the interviews and site visits. The codebook fostered team members' ability to consistently label or code segments of the narrative data.
Evaluation team debriefings The evaluation team engaged in a multistage process of data debriefings that were instrumental in verifying and communicating interview content, facilitating reflection and personal reflexivity, and synthesizing data findings according to evaluation topic. The types and content of the peer debriefings are outlined below:
- Individual Interview Debriefings
- ο
Using the co-facilitator's end-of-interview summary as a basis, interview team members' documented interviewees' key points or messages, reflected on the interview process, engaged in and acknowledged personal reflexivity, and participated in a preliminary analysis and interpretation of the data collected during the given individual in-depth interview or group interview.
- Daily or Every-2-Day Interview Debriefings
- ο
All delegation team members convened to share key points that emerged from the interviews of which they were a part, their perspectives about and interpretations regarding the data, and their personal reflections/reflexivity.
- Synthesis (End of Week 1)
- ο
All delegation team members engaged in a midpoint synthesis of interview findings, the process of which was facilitated by the team leader and structured according to evaluation topic. To assist with the synthesis, each team member received a copy of the interview debriefings that had been conducted so far on the country visit.
- ο
Committee members often participated during the first week of the 2-week country visits. The synthesis process at the end of Week 1 was thus critical in eliciting committee member insights into country visit data and interpretation before they exited the country.
- Exit Synthesis (End of Week 2)
- ο
All delegation team members still in country engaged in an 8- to 10-hour process of verbally synthesizing the findings associated with data collection prior to exiting the country. As with the synthesis process at the end of Week 1, each team member received a copy of the interview debriefings that had been conducted. The team leader facilitated the exit synthesis process, which was structured according to evaluation topics and included data documentation, reflection and reflexivity regarding the data collected, and verbal analysis and interpretation notations.
- Across-Country Debriefings and Discussions
- ο
Periodically, between clusters of country visits, IOM study staff participated in a discussion and synthesis of the qualitative findings according to evaluation topic and identified consonance or differences in these findings across a number of countries.
- ο
At committee meetings that occurred periodically between clusters of country visits, committee members, either as a whole committee or in working groups focused on specific content areas, participated in discussions of the analysis and interpretation of interview data, including review of draft data presentations.
Accuracy of data collection Accuracy was critical in documenting the data collected for this evaluation. With participants' permission, interviews were digitally recorded in conjunction with handwritten notes taken by members of the interview team. Professional transcriptionists ultimately transcribed the digitally recorded interviews, but the need for timeliness, efficiency, ease of comprehension, and engagement in data analysis from the onset of data collection led the evaluation team to rely on their own typed transcription of handwritten interview notes as the primary source of interview data for analysis. To ensure completeness and accuracy of these interview notes, interview team members engaged in an independent, detailed review of the note-taker's transcribed handwritten notes. This process involved an initial draft by the assigned note taker, a review by another team member who participated in the interview, and a final resolution round by the original note taker.
During the end-of-interview summary provided by the co-facilitator (or the facilitator when there was no co-facilitator), interviewees addressed the accuracy of the main end-of-interview points that the co-facilitator shared with them by affirming, correcting, or adding to the end-of-interview summary. In addition, the interview team debriefed each interview shortly after it occurred to affirm the accurate documentation of main points using the co-facilitator's potentially revised summary as a foundation and contributing additional details. An additional accuracy check was afforded team members who could reference the digital recording of the interview when clarifying segments of narrative or resolving issues of disagreement regarding the content of a particular interview.
Collection of Non-Country Visit Interview Data
As part of the data collection effort for the evaluation, IOM staff and consultants also conducted a series of 32 non-country visit interviews with key stakeholders. The interviewees included the USG at PEPFAR HQ level (including OGAC, CDC, and USAID), U.S.-based implementing partners at HQ level, and other organizations that work in the global response to HIV, including multilateral organizations, NGOs, and another bilateral donor. As with the country visit interviewees, non-country visit interviewees were not only selected through purposeful sampling, but also prioritized on the basis of targeted focus areas within the evaluation, and the process of mapping data sources for evaluation questions. Semi-structured interviews were conducted using the same methodology as the country visit interviews, using interview guides with questions and prompts adapted as appropriate for each interview.
Analyses of Qualitative Data
In-country data analysis process In-country data review and preliminary analysis occurred at various levels and at several times during country visits, during the debriefings and synthesis discussions described previously. As soon as possible after each interview, team members conducted a post-interview debriefing to discuss and document the main points shared by the interviewee(s). Delegation members also convened routinely as an entire team during the country visit to engage in debriefings to share with each other the main points from the data across all the interviews that were conducted.
At the close of the first week of each 2-week country visit and again at the close of the country visit, the team conducted, respectively, an end-of-week debriefing and an exit synthesis debriefing that utilized an inductive analysis approach for the purpose of identifying dominant themes that emerged from the data. Both of these processes began with team members individually reading the debriefing notes from interviews conducted during the week to review key data from the interviews and to identify concepts and themes emerging from the data. Delegation members then collectively discussed the data and dominant themes that arose from the interviews, systematically using categories that were pre-selected based on the evaluation objectives. The delegation team differentiated between evidence or the responses heard during the interviews, and analysis and interpretation, which reflected the delegation's interpretations of what the evidence meant, focusing on the meaning in relation to the evaluation objectives. The output from these processes was an exit synthesis document capturing the key evidence and analysis and interpretation from the interviews grouped by evaluation category, and a key messages document capturing the main themes that emerged across the interviews.
These documents were then included as part of the country visit summary, which was reviewed by the members of the trip delegation and then posted on the committee portal. The country visit summary also includes other information provided to the delegation in advance of the trip in the form of a country brief, including background research on the country context and PEPFAR program as well as basic financial data and OGAC and other programmatic or indicator data (including UNAIDS data). The country visit summary is a compilation of the data from these multiple sources, not a triangulated analysis of the data and evidence available for each country. The goal was to provide a “snapshot” overview to inform the rest of the committee about the visit and the country and to provide a centralized source for country data.
Synthesis of exit syntheses To provide the committee with a sense of the overall current findings emerging from the interview data, for some of the evaluation categories the IOM staff and consultant conducted a synthesis to identify and present the dominant themes that emerged in the exit syntheses across countries. This synthesis was conducted and presented in a variety of ways, ranging from an analytical synthesis presented in narrative form to data grouped in bulleted form by sub-themes, which offered less synthesis and analysis but was closer to the “raw” data.
Additional analysis of interview data Additional data summaries, syntheses, and analyses from both the country visit and non-country visit interview data were generated using methods detailed below.
Members of the IOM staff used NVivo software (version 9.0) to conduct macro-level coding of the data using detailed interview notes generated by IOM staff and consultants or transcripts generated by contracted professional transcriptionists from audio-recordings of interviews. The subset of data coded in NVivo comprised more than half of the interviews, purposefully selected for representation across countries and stakeholder types. This coding was based on a standardized project code book with each code reflecting important data concepts with inclusion and exclusion criteria. The data concepts represented in the codebook were based on evaluation topics identified in the evaluation planning phase (IOM and NRC, 2010), the evaluation committee's development of priority evaluation questions, and the exit synthesis process and review of initial data collected from the pilot country visits and other early country visits. For synthesis and analysis, these coded data were separated and extracted by querying for a single code or combinations of the macro-level codes across interviews. In some cases, data were also extracted from the NVivo dataset using targeted word search queries.
Building on this initial thematic identification, IOM staff or Dr. Knight then conducted a more in-depth and refined analysis through repeated reading, reflection, and continued micro-level coding of the data for narrower subconcepts. This led to inductive identification of themes, patterns, and categories that emerged as findings from the data. This was followed by deductive confirmation and disconfirmation of those findings and determinations of data saturation for topics and themes (i.e., whether any new data had emerged). Prolonged engagement in data collection also led team members to affirm data saturation. Delegation evaluation teams recognized data saturation, through multiple iterations of individual and group analyses and discussions described below, as the repetition of information to the point of redundancy, which indicated that data collection could be reasonably concluded (Merriam, 2002, 2009; Patton, 2002).
In the next iteration of the analytical process, drafts of data analysis outputs were read for discussion and revision by members of the project staff, consultants, and evaluation committee members who were familiar with the interview data and had participated in data collection and in-country data analyses. In addition, interview debriefing and exit synthesis documents from all interviews, including those not in the initial coded dataset, were used to carry out supplementary deductive confirmation and disconfirmation of findings that emerged from the coded data, and to identify specific additional interview notes and transcripts for enrichment of the analysis of the coded data.
These interview data findings and analyses were presented in a number of ways, including in narrative form with accompanying illustrative quotations, in summary tables, or in bulleted groupings by subconcepts. The presentation of quotes was used when one person's words provided a memorable description of an issue that was resonant with multiple interviewees or perspectives, or in some cases when one person's words represented a meaningful disconfirming perspective. For this report, single quotation marks were used to denote an interviewee's perspective with wording extracted from transcribed notes written during the interview, and double quotation marks were used to denote an exact quote from an interviewee either confirmed by listening to the audio-recording of the interview or extracted from a full transcript of the audio-recording.
Interview data presented in the report are accompanied by a citation key. Interviews in qualitative research are often cited with a brief descriptive demographic phrase; however, this was not feasible for an evaluation of this scope, with more than 400 interviews and the frequent citations for multiple interviews. Therefore, a citation tag was developed to allow the reader to identify the key characteristics relevant for the analysis and interpretation of the data for this evaluation, including the range of countries and interviews represented and the stakeholder type. The interview citation key is shown in Box C-1.
![Box Icon](/corehtml/pmc/css/bookshelf/2.26/img/box-icon.gif)
BOX C-1
Interview Citation Key. Country Visit Exit Synthesis Key: Country # + ES Country Visit Interview Citation Key: Country # + Interview # + Organization Type
REFERENCES
- Bendavid E, Bhattacharya J. The President's Emergency Plan for AIDS Relief in Africa: An evaluation of outcomes. Annals of Internal Medicine. 2009;150(10):688–695. [PMC free article: PMC2892894] [PubMed: 19349625]
- Bendavid E, Holmes CB, Bhattacharya J, Miller G. HIV development assistance and adult mortality in Africa. Journal of the American Medical Association. 2012;307(19):2060–2067. [PMC free article: PMC3434229] [PubMed: 22665105]
- Bouey P, De Leon J. Interview; April 27, 2011. Washington, DC: 2011.
- Central Intelligence Agency. The world factbook. n.d. [June 15, 2012]. https://www
.cia.gov/library /publications/download . - CGD (Center for Global Development). PEPFAR funding data full dataset. Washington, DC: CGD; 2008.
- Creswall JW. Qualitative inquiry and research design: Choosing among five approaches. Thousand Oaks, CA: SAGE; 2007.
- Institute for Health Metrics and Evaluation. Global health data exchange. n.d. [June 15, 2012]. http://www
.healthmetricsandevaluation .org /ghdx/record/adult-mortality-estimates-country-1970-2010 . - International epidemiologic Databases to Evaluate AIDS. International epidemiologic databases to evaluate AIDS. n.d. [March 17, 2011]. http://iedea
.org . - IOM and NRC (Institute of Medicine and National Research Council). Strategic approach to the evaluation of programs implemented under the Tom Lantos and Henry J. Hyde U.S. Global Leadership Against HIV/AIDS, Tuberculosis, and Malaria Reauthorization Act of 2008. Washington, DC: The National Academies Press; 2010.
- McCullough R, Miller L. From the ground up: Building comprehensive HIV/AIDS care programs in resource-limited settings. Washington, DC: Elizabeth Glaser Pediatric AIDS Foundation; 2009. Surveying the global HIV/AIDS landscape. In.
- Merriam SB. Assessing and evaluating qualitative research. Edited by Sharon B. Merriam and Associates, Qualitative research in practice. San Francisco: Josey-Bass; 2002.
- Merriam SB. Qualitative research: A guide to design and implementation. San Francisco: Jossey-Bass; 2009.
- OECD (Organisation for Economic Co-operation and Development). Paris: OECD; OECD.Stat. 2012 database.
- OGAC (Office of the U.S. Global AIDS Coordinator). Emergency Plan for AIDS Relief fiscal year 2005 operational plan: June 2005 update. Washington, DC: OGAC; 2005a.
- OGAC. The President's emergency plan for AIDS relief: Indicators, reporting requirements, and guidelines for focus countries. Washington, DC: OGAC; 2005b.
- OGAC. The President's Emergency Plan for AIDS Relief. FY 2007 supplemental COP guidance resource guide. Washington, DC: OGAC; 2006a.
- OGAC. The President's emergency plan for AIDS relief: FY 2007 country operational plan guidance. Washington, DC: OGAC; 2006b.
- OGAC. The U.S. President's Emergency Plan for AIDS Relief Fiscal Year 2006: Operational plan. 2006 August update. Washington, DC: OGAC; 2006c.
- OGAC. The President's emergency plan for AIDS relief: Indicators, reporting requirements, and guidelines. Indicators reference guide: FY 2007 reporting/FY 2008 planning. Washington, DC: OGAC; 2007a.
- OGAC. The U.S. President's Emergency Plan for AIDS Relief fiscal year 2007: Operational plan. 2007 June update. Washington, DC: OGAC; 2007b.
- OGAC. The U.S. President's Emergency Plan for AIDS Relief (PEPFAR) fiscal year 2008: PEPFAR operational plan. June 2008. Washington, DC: OGAC; 2008.
- OGAC. Guidance for PEPFAR partnership frameworks and partnership framework implementation plans. Washington, DC: OGAC; 2009a. (Version 2.0).
- OGAC. The President's Emergency Plan for AIDS Relief: Next generation indicators reference guide. Version 1.1. Washington, DC: OGAC; 2009b.
- OGAC. The U.S. President's Emergency Plan for AIDS Relief (PEPFAR) fiscal year 2009: PEPFAR operational plan. November 2010. Washington, DC: OGAC; 2010.
- OGAC. PEPFAR response to data request from IOM for the PEPFAR II evaluation. Washington, DC: OGAC; 2011a.
- OGAC. The U.S. President's Emergency Plan for AIDS Relief (PEPFAR) fiscal year 2010: PEPFAR operational plan. Washington, DC: OGAC; 2011b.
- OGAC. The U.S. President's Emergency Plan for AIDS Relief (PEPFAR) fiscal year 2011: PEPFAR operational plan. Washington, DC: OGAC; 2011c.
- OGAC. Country operational plans. n.d.-a. [April 9, 2012]. http://www
.pepfar.gov /countries/cop/index.htm . - OGAC. Obligation and outlay reports. n.d.-b. [November 29, 2012]. http://www
.pepfar.gov/about/c24880.htm . - OGAC. Operational plans. n.d.-c. [April 9, 2012]. http://www
.pepfar.gov/about/c19388.htm . - OGAC. Partners. n.d.-d. [July 31, 2012]. http://www
.pepfar.gov /funding/budget/partners/index.htm . - Oomman N, Bernstein M, Rosenzweig S. The numbers behind the stories: PEPFAR funding for fiscal years 2005 to 2006. Washington, DC: CGD; 2008.
- Patton MQ. Qualitative research and evaluation methods. 3rd. Thousand Oaks, CA: SAGE; 2002.
- Rajaratnam JK, Marcus JR, Levin-Rector A, Chalupka AN, Wang H, Dwyer L, Costa M, Lopez AD, Murray CJ. Worldwide mortality in men and women aged 15-59 years from 1970 to 2010: A systematic analysis. Lancet. 2010;375(9727):1704–1720. [PubMed: 20434763]
- Rossman GB, Rallis S. Learning in the field: An introduction to qualitative research. Thousand Oaks, CA: SAGE; 2012.
- Sessions M. Overview of the President's Emergency Plan for AIDS Relief (PEPFAR). Washington, DC: CGD; 2006.
- U.S. Department of Commerce. International data base population estimates and projections methodology. 2010. [March 15, 2011]. http://www
.census.gov /population/international /data/idb/estandproj.pdf . - U.S. Department of Commerce. United States census bureau: International database. n.d. [March 15, 2011]. http://www
.census.gov /population/international /data/idb/informationGateway.php . - UNAIDS (Joint United Nations Programme on HIV/AIDS). Monitoring the declaration of commitment on HIV/AIDS: Guidelines on construction of core indicators: 2010 reporting. Geneva: UNAIDS; 2009.
- UNAIDS. UNAIDS AIDSinfo database. 2011. [March 15, 2011]. http://www
.aidsinfoOnline.org . - UNAIDS. Data tools. n.d. [June 15, 2012]. http://www
.unaids.org /en/dataanalysis/datatools . - UNICEF (United Nations Children's Fund). The state of the world's children 2011: Adolescence, an age of opportunity data. 2011. [March 14, 2011]. http://www
.unicef.org /sowc2011/statistics.php . - United Nations, Department of Economic and Social Affairs. World population prospects. n.d. [June 15, 2012]. http://esa
.un.org/wpp /Excel-Data/mortality.htm . - USAID (U.S. Agency for International Development). MEASURE DHS: Demographic and health surveys. 2011. [March 15, 2011]. http://www
.measuredhs.com . - USAID. Data Online for population, health, and nutrition. n.d. [March 17, 2011]. http://dolphn
.aimglobalhealth.org . - WHO (World Health Organization). Global atlas of the world health workforce. n.d.-a. [March 17, 2011]. http://apps
.who.int/globalatlas/default .asp . - WHO. Global health observatory data repository. n.d.-b. [March 15, 2011]. http://apps
.who.int/gho/data/node.main . - WHO. WHO global health workforce statistics. n.d.-c. [March 17, 2012]. http://www
.who.int/hrh /statistics/hwfstats/en . - WHO. WHO mortality database. n.d.-d. [June 15, 2012]. http://www
.who.int/healthinfo /mortality_data/en/index.html . - Wolf ZR. Exploring the audit trail for qualitative investigations. Nurse Educator. 2003;28(4):175–178. [PubMed: 12878896]
- World Bank. World data bank: World development indicators. n.d. [March 15, 2011]. http://databank
.worldbank .org/ddp/home.do .
Footnotes
- 1
To represent the greatest intensity of PEPFAR's investment, the scope of this evaluation was defined to focus on the 31 partner countries submitting an annual Country Operational Plan (COP) at the time of the initiation of the planning phase for this evaluation in 2009. They include the original 15 focus countries (Botswana, Republic of Côte d'Ivoire, Federal Democratic Republic of Ethiopia, Cooperative Republic of Guyana, Republic of Haiti, Republic of Kenya, Republic of Mozambique, Republic of Namibia, Federal Republic of Nigeria, Republic of Rwanda, Republic of South Africa, United Republic of Tanzania, Republic of Uganda, Socialist Republic of Vietnam, and Republic of Zambia), as well as the following additional countries: Republic of Angola, Kingdom of Cambodia, People's Republic of China, Democratic Republic of the Congo, Dominican Republic, Republic of Ghana, Republic of India, Republic of Indonesia, Kingdom of Lesotho, Republic of Malawi, Russian Federation, Republic of the Sudan, Kingdom of Swaziland, Kingdom of Thailand, the Ukraine, and the Republic of Zimbabwe.
- 2
SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries.® indicates USA registration.
- 3
No indicator guidance was made available for indicators reported during FY 2004 and FY 2005; for this evaluation the FY 2006 guidance definitions were referenced for interpretation of FY 2004 and FY 2005 indicators.
- 4
OGAC defines these 2009 classifications in the NGI guidance as follows (OGAC, 2009b): Essential/Reported to HQ: Indicators that are aggregated and reported to PEPFAR headquarters.
- 5
Track 1.0 Partners in this report refers to the four partners that were the primary large-scale implementers of ART in PEPFAR's centrally funded Track 1.0 program. These partners also implemented other HIV services and programs, and there were also other centrally funded Track 1.0 partners in other program areas (OGAC, 2006a).
Essential/Not Reported to HQ: Indicators that do not need to be aggregated and reported to PEPFAR headquarters; however, partners are required to report applicable indicators to the PEPFAR country teams. In addition, PEPFAR country teams are expected to support and encourage intermittent surveillance required to monitor indicators not routinely captured through programs. The intent of these indicators is to highlight critical program areas that PEPFAR country teams should be monitoring and to provide teams increased flexibility to work within the context of the national system.
Recommended: These are additional indicators for partners and program managers who need information for program management beyond the minimum set reported to OGAC Headquarters. The PEPFAR interagency technical working groups selected and recommended these indicators as important areas for program managers to monitor, but they are not considered indispensable to program tracking. The intent of these indicators is to encourage comprehensive monitoring of programs, provide additional recommendations on indicators, and give PEPFAR country teams increased flexibility to work within the context of the national system. These indicators are not subject to audit.
- Evaluation Methods - Evaluation of PEPFAREvaluation Methods - Evaluation of PEPFAR
- TonB-dependent receptor [Chryseobacterium indologenes]TonB-dependent receptor [Chryseobacterium indologenes]gi|736732558|ref|WP_034736573.1|Protein
- protein ABHD13 [Mus musculus]protein ABHD13 [Mus musculus]gi|299473802|ref|NP_081144.1|Protein
Your browsing activity is empty.
Activity recording is turned off.
See more...