ACCURACY OF THE USDA METHODOLOGY

In preparing a budget request for the WIC program, USDA employs the most current survey data from the March Current Population Survey (CPS). As discussed in the previous chapter, the survey data for a given year could be used to predict eligibility and participation at least four years into the future. For example, when USDA was preparing their budget request for 2002, they would have employed data covering calendar year 1998. The accuracy of this forecast of the number of participants in 2002 depends on two factors. The first is how well the methodology predicts the number of participants in the year for which there are data (in this example, 1997). The second is the validity of the assumption that participation will be unchanged over the forecast period.

To assess the level of prediction error in the estimates—the error resulting from the use of currently available data to predict future eligibility and participation—Table 3-1 compares the estimated number of participants by category with actual administrative counts of participants. This ratio was computed using data from 1992–2000. Estimates are first totaled over all categories and then given separately by each eligibility category.¹

The first line of each category contains the ratio of the estimated number of WIC participants in a given year, t, to the actual number of WIC participants in the same year, t (labeled concurrent year ratio). Examining the first set of ratios, which are totaled over all categories, we see that the ratio of the estimated to actual number of participants ranges from 1.37 in 1992 to 0.86 in 2000. According to this measure of accuracy, the estimates of the number of participants were overstated by 37 percent in 1992 but understated by 14 percent in 2000. Patterns within eligibility categories vary greatly, however. The ratios for infants and breastfeeding postpartum women show a consistent underestimate of the number of WIC participants compared with administrative counts of actual participants. This underestimate is getting worse over time. For infants, the ratios range from about 0.80 in 1992 to about 0.61 in 2000. For breastfeeding postpartum women, the ratios range from 0.99 in 1992 to about 0.49 in 2000. Ratios for children show that the estimates initially overestimate the number of participants but are very close to the actual number of participants from 1998 onward. Estimates for pregnant women were overstated in early years but in subsequent years have been slightly understated. For nonbreastfeeding postpartum women, estimates of the number of participants are significantly understated for every year except 1992 and 1993.

These ratios do not really reflect the task USDA confronts each year in trying to predict eligibility and participation for the year the proposed budget is to cover. In actuality, estimates of the numbers eligible and likely to participate for a given year, t, are used to predict the number of participants forecast for year t + 4 (e.g., data from 1992 are used to predict FY 1996 participants). To take the forecasting component of the estimation process into account, the estimated number of participants in year t are compared with the actual number of participants from administrative records from year t + 4 for each eligibility category and overall categories (labeled forecasted ratio). These ratios are given in the last line of each eligibility category.

TABLE 3-1

Accuracy of Current Methods Used to Predict WIC Participants: Ratios of Estimated Participants to Actual Participants, 1992–2000.

Based on the forecasted ratios for total numbers of participants, it appears that the current USDA method of estimating the number of participants for a future budget cycle is quite accurate. The total number of predicted participants matches the number of actual participants quite closely, ranging from a slight overestimate in 1996 (2.6 percent) to a slight underestimate in 2000 (4.1 percent), indicating very small levels of errors. These smaller error levels are really just a coincidence, because the ratios for the total number of participants mask substantial over- and underestimation among the different eligibility categories. Numbers of infant participants are substantially underestimated across all years, as are estimates of the number of breastfeeding postpartum women. In contrast, numbers of participating children are overestimated by 14 to 21 percent compared with administrative records. Numbers of participating pregnant women are overestimated also, but the degree is not so serious in later years. Numbers of nonbreastfeeding postpartum women are consistently underestimated and become more substantially underestimated in later years, so that by 2000, estimates are understated by about 17 percent.

Although USDA's current methods to estimate eligibility and participation seem to accurately forecast the overall number of participants, these methods do not accurately forecast specific eligibility categories. Accurately estimating the number of participants in each category is important because food package costs differ across each eligibility category. Furthermore, changes in program rules or program administration could affect eligibility and participation for each category differently. For example, increases in the Medicaid income thresholds in states for infants and children would affect eligibility of those groups but not that of pregnant and postpartum women.

SOURCES OF ERROR

The true number of eligible persons who are likely to participate is unknown. In making budgetary and programmatic decisions, USDA's goal is to come as close to the true number of eligibles and participants as possible. There are two possible sources of error in making these estimates: (1) errors that cause a systematic bias in the estimated number of persons eligible or likely to participate and (2) prediction errors.

EVALUATING CURRENT AND ALTERNATIVE METHODS FOR ESTIMATING ELIGIBILITY AND PARTICIPATION

In this report we evaluate methods currently used to estimate eligibility and participation for WIC and examine alternatives to current methods. We outline different methodological options and include recommendations for new approaches to the various components of the estimation methodology. The panel considered several factors for evaluating alternatives:

• The accuracy of the estimates.
• The feasibility of implementing an approach (e.g., the expense and burden of implementing an approach).
• The quality, availability, and timeliness of data used by an approach.
• Correspondence of the method with current WIC rules and their application at the local level. For example, adjunctive eligibility is part of the rules of the program, and methods for estimating eligibility should account for it.

For some components of the estimation process, it is not clear how to assess the accuracy of an approach. For example, the minor adjustment currently made to account for adjunctive eligibility appears to be inadequate (National Research Council, 2001). However, data limitations create problems in determining the true number of people who gain eligibility for WIC only through adjunctive eligibility. Estimates based on survey data are limited because respondents underreport Medicaid and other program participation (although to what degree is not known). Administrative data on Medicaid enrollees do not identify the age of children nor the income of enrollees; therefore, no administrative data on the number of people who would gain eligibility for WIC solely through adjunctive eligibility are available. The panel also based its assessments of methodologies on the premise that the methodology should reflect current rules and practices of the program.

Finally, the panel recognizes that the USDA estimates serve different purposes and that different methods of estimation may be appropriate for different purposes. Evaluation of the effectiveness of the WIC program requires estimates of eligibility and participation. Because eligibility is not directly observed, estimates must be based on information reported from survey data. To make budgetary forecasts, reasonably accurate estimates might be forecasted based on administrative records from past years. Chapters 4–7 focus on the estimation of numbers of people eligible for WIC. Chapter 8 focuses on estimates of participation among eligible people and methods for estimating participation among them. In Chapter 9, we synthesize all our findings and provide different options for estimating eligibility and participation.

SUMMARY

This chapter began with an assessment of the accuracy of the USDA estimates to predict the number of WIC participants. Using available data to examine the accuracy of participation estimates, the panel found that the total number of participants matches the reported number quite closely. However, the numbers of participating infants and breastfeeding postpartum women are seriously underestimated, while the numbers of participating children and pregnant women are overestimated. The chapter also explains the two types of errors the panel investigated in its review of the current methodology—errors that cause systematic bias and prediction errors. Finally, in evaluating estimation methods, the panel considered accuracy, feasibility, characteristics of data sources, and correspondence of the method with current rules and their application at the local level.

Footnotes

1

Administrative records on the number of participants may include ineligible participants. Estimates of eligibility may differ from administrative records because of this, but we do not know how big these differences may be.