NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.
National Research Council (US) Panel to Evaluate the USDA's Methodology for Estimating Eligibility and Participation for the WIC Program; Ver Ploeg M, Betson DM, editors. Estimating Eligibility and Participation for the WIC Program: Final Report. Washington (DC): National Academies Press (US); 2003.
Estimates of the number of people eligible and likely to participate in WIC are useful for both budgetary and evaluative needs only to the extent to which they are reasonably accurate representations of their “true” levels and trends in eligibility and participation. It is not possible to observe actual eligibility for WIC in the population because eligibility is only observed when an individual applies for WIC benefits. However, administrative records can be utilized to construct a benchmark for examining the accuracy of the estimates of the number of participants.
This chapter begins by accessing the accuracy of the USDA method used to predict the number of women, infants, and children who participate in the program. For the overall population, USDA's methodology has led to relatively accurate forecasts of the number of participants. However, for the various subgroups of WIC eligible participants—women, infants and children—the accuracy of their forecasts of the total number has been poor. The chapter goes on to identify potential sources of error.
ACCURACY OF THE USDA METHODOLOGY
In preparing a budget request for the WIC program, USDA employs the most current survey data from the March Current Population Survey (CPS). As discussed in the previous chapter, the survey data for a given year could be used to predict eligibility and participation at least four years into the future. For example, when USDA was preparing their budget request for 2002, they would have employed data covering calendar year 1998. The accuracy of this forecast of the number of participants in 2002 depends on two factors. The first is how well the methodology predicts the number of participants in the year for which there are data (in this example, 1997). The second is the validity of the assumption that participation will be unchanged over the forecast period.
To assess the level of prediction error in the estimates—the error resulting from the use of currently available data to predict future eligibility and participation—Table 3-1 compares the estimated number of participants by category with actual administrative counts of participants. This ratio was computed using data from 1992–2000. Estimates are first totaled over all categories and then given separately by each eligibility category.1
The first line of each category contains the ratio of the estimated number of WIC participants in a given year, t, to the actual number of WIC participants in the same year, t (labeled concurrent year ratio). Examining the first set of ratios, which are totaled over all categories, we see that the ratio of the estimated to actual number of participants ranges from 1.37 in 1992 to 0.86 in 2000. According to this measure of accuracy, the estimates of the number of participants were overstated by 37 percent in 1992 but understated by 14 percent in 2000. Patterns within eligibility categories vary greatly, however. The ratios for infants and breastfeeding postpartum women show a consistent underestimate of the number of WIC participants compared with administrative counts of actual participants. This underestimate is getting worse over time. For infants, the ratios range from about 0.80 in 1992 to about 0.61 in 2000. For breastfeeding postpartum women, the ratios range from 0.99 in 1992 to about 0.49 in 2000. Ratios for children show that the estimates initially overestimate the number of participants but are very close to the actual number of participants from 1998 onward. Estimates for pregnant women were overstated in early years but in subsequent years have been slightly understated. For nonbreastfeeding postpartum women, estimates of the number of participants are significantly understated for every year except 1992 and 1993.
These ratios do not really reflect the task USDA confronts each year in trying to predict eligibility and participation for the year the proposed budget is to cover. In actuality, estimates of the numbers eligible and likely to participate for a given year, t, are used to predict the number of participants forecast for year t + 4 (e.g., data from 1992 are used to predict FY 1996 participants). To take the forecasting component of the estimation process into account, the estimated number of participants in year t are compared with the actual number of participants from administrative records from year t + 4 for each eligibility category and overall categories (labeled forecasted ratio). These ratios are given in the last line of each eligibility category.
TABLE 3-1Accuracy of Current Methods Used to Predict WIC Participants: Ratios of Estimated Participants to Actual Participants, 1992–2000
| 1992 | 1993 | 1994 | |
|---|---|---|---|
| All Eligibility Categories | |||
| Concurrent year ratios: a | 1.365 | 1.300 | 1.111 |
| Forecasted ratios: b | |||
| Infants | |||
| Concurrent year ratios: | 0.795 | 0.790 | 0.693 |
| Forecasted ratios: | |||
| Children | |||
| Concurrent year ratios: | 1.756 | 1.642 | 1.385 |
| Forecasted ratios: | |||
| Pregnant Women | |||
| Concurrent year ratios: | 1.270 | 1.115 | |
| Forecasted ratios: | |||
| Nonbreastfeeding Women | |||
| Concurrent year ratios: | 1.351 | 1.003 | |
| Forecasted ratios: | |||
| Breastfeeding Women | |||
| Concurrent year ratios: | 0.993 | 0.770 | |
| Forecasted ratios: | |||
| All Postpartum Women | |||
| Concurrent year ratios: | 1.218 | 0.919 | |
| Forecasted ratios: | |||
- a
Concurrent ratios = Estimated Participants in year t / Actual Participants in year t.
- b
Forecasted Ratios = Estimated Participants for year (t + 4) / Actual Participants year t.
Note: To compute the number of eligible persons who will participate, an 80 percent participation rate is assumed.
| 1995 | 1996 | 1997 | 1998 | 1999 | 2000 |
|---|---|---|---|---|---|
| 1.020 | 0.961 | 0.908 | 0.865 | 0.842 | 0.860 |
| 1.026 | 1.040 | 0.976 | 0.962 | 0.959 | |
| 0.698 | 0.674 | 0.630 | 0.602 | 0.589 | 0.609 |
| 0.732 | 0.738 | 0.657 | 0.668 | 0.650 | |
| 1.197 | 1.117 | 1.069 | 1.023 | 0.996 | 1.014 |
| 1.180 | 1.205 | 1.179 | 1.141 | 1.167 | |
| 1.127 | 1.070 | 0.998 | 0.953 | 0.953 | 0.987 |
| 1.164 | 1.170 | 1.039 | 1.082 | 1.054 | |
| 0.932 | 0.845 | 0.796 | 0.774 | 0.755 | 0.781 |
| 0.918 | 0.933 | 0.844 | 0.857 | 0.834 | |
| 0.732 | 0.664 | 0.560 | 0.516 | 0.485 | 0.487 |
| 0.722 | 0.656 | 0.563 | 0.550 | 0.520 | |
| 0.861 | 0.780 | 0.705 | 0.672 | 0.646 | 0.660 |
| 0.848 | 0.827 | 0.733 | 0.733 | 0.705 |
Actual number of participants come from FNS administrative records. Estimated numbers of participants are the USDA estimations given to the panel by USDA. The forecasted ratios assume that estimates of participation are used to predict the actual level of participants four years into the future.
Based on the forecasted ratios for total numbers of participants, it appears that the current USDA method of estimating the number of participants for a future budget cycle is quite accurate. The total number of predicted participants matches the number of actual participants quite closely, ranging from a slight overestimate in 1996 (2.6 percent) to a slight underestimate in 2000 (4.1 percent), indicating very small levels of errors. These smaller error levels are really just a coincidence, because the ratios for the total number of participants mask substantial over- and underestimation among the different eligibility categories. Numbers of infant participants are substantially underestimated across all years, as are estimates of the number of breastfeeding postpartum women. In contrast, numbers of participating children are overestimated by 14 to 21 percent compared with administrative records. Numbers of participating pregnant women are overestimated also, but the degree is not so serious in later years. Numbers of nonbreastfeeding postpartum women are consistently underestimated and become more substantially underestimated in later years, so that by 2000, estimates are understated by about 17 percent.
Although USDA's current methods to estimate eligibility and participation seem to accurately forecast the overall number of participants, these methods do not accurately forecast specific eligibility categories. Accurately estimating the number of participants in each category is important because food package costs differ across each eligibility category. Furthermore, changes in program rules or program administration could affect eligibility and participation for each category differently. For example, increases in the Medicaid income thresholds in states for infants and children would affect eligibility of those groups but not that of pregnant and postpartum women.
SOURCES OF ERROR
The true number of eligible persons who are likely to participate is unknown. In making budgetary and programmatic decisions, USDA's goal is to come as close to the true number of eligibles and participants as possible. There are two possible sources of error in making these estimates: (1) errors that cause a systematic bias in the estimated number of persons eligible or likely to participate and (2) prediction errors.
Errors Causing Systematic Biases
Errors may arise because data or methods used to make the estimates are not able to fully capture all the programmatic features or the realities of individuals' economic and family situations, leading to inaccurate estimates of eligibility and participation. For example, the March CPS, the data set used currently to make eligibility estimates, collects annual income instead of monthly income. As Chapter 2 explains, local WIC agencies can use weekly and monthly income, rather than annual income, to determine whether an applicant is eligible for WIC. Variation in monthly income could mean that some people gain eligibility for WIC in some months but have annual incomes that would otherwise make them ineligible for WIC. The following possible sources of systematic error are identified and discussed in this report:
- The undercount of infants and the overcount of children in the CPS.
- The use of annual rather than monthly income in estimating eligibility for WIC.
- Not fully accounting for adjunctive eligibility through means-tested programs, particularly Medicaid.
- The inaccuracy of adjustments to account for nutritional risk among income-eligible persons.
- The inaccuracy of adjustments to account for breastfeeding status among postpartum women.
- The inaccuracy of adjustments for the percentage of eligible persons who will participate in WIC.
Prediction Errors
The second source of errors arises because eligibility and participation must be predicted for future years from data that are probably four years old. For example, the following changes could lead to prediction errors:
- Demographic changes (e.g., lower birth rates or increased immigration).
- Changes in family structure (e.g., if the proportion of infants and children living in single-parent rather than two-parent families declines, then, since single-parent families are, on average, poorer than two-parent families, the proportion of those income-eligible infants and children should decline).
- Changes in the income distribution due to changes in wages or unemployment.
- Changes in program rules, including WIC, but also in other means-tested programs for which adjunctive eligibility is granted. Medicaid rule changes in recent years are a primary example.
- Changes in WIC program administrative practices.
- Changes in participation rates of other means-tested programs.
- The length of time between the year that data are available for esti mation and the year for which the estimate is being made. All else equal, a longer time period should lead to larger errors.
These changes could, of course, have opposite effects on estimates of eligibility and participation. For example, the increase in Medicaid income eligibility thresholds would tend to increase the number of eligible persons. However, the economic expansion of the late 1990s should have led to a decrease in the number of persons income eligible for WIC and for Medicaid. It is difficult to infer the extent to which these changes offset each other or not.
In considering current and alternative approaches to estimating eligibility and participation for WIC, the panel has attempted to address these two sources of errors.
EVALUATING CURRENT AND ALTERNATIVE METHODS FOR ESTIMATING ELIGIBILITY AND PARTICIPATION
In this report we evaluate methods currently used to estimate eligibility and participation for WIC and examine alternatives to current methods. We outline different methodological options and include recommendations for new approaches to the various components of the estimation methodology. The panel considered several factors for evaluating alternatives:
- The accuracy of the estimates.
- The feasibility of implementing an approach (e.g., the expense and burden of implementing an approach).
- The quality, availability, and timeliness of data used by an approach.
- Correspondence of the method with current WIC rules and their application at the local level. For example, adjunctive eligibility is part of the rules of the program, and methods for estimating eligibility should account for it.
For some components of the estimation process, it is not clear how to assess the accuracy of an approach. For example, the minor adjustment currently made to account for adjunctive eligibility appears to be inadequate (National Research Council, 2001). However, data limitations create problems in determining the true number of people who gain eligibility for WIC only through adjunctive eligibility. Estimates based on survey data are limited because respondents underreport Medicaid and other program participation (although to what degree is not known). Administrative data on Medicaid enrollees do not identify the age of children nor the income of enrollees; therefore, no administrative data on the number of people who would gain eligibility for WIC solely through adjunctive eligibility are available. The panel also based its assessments of methodologies on the premise that the methodology should reflect current rules and practices of the program.
Finally, the panel recognizes that the USDA estimates serve different purposes and that different methods of estimation may be appropriate for different purposes. Evaluation of the effectiveness of the WIC program requires estimates of eligibility and participation. Because eligibility is not directly observed, estimates must be based on information reported from survey data. To make budgetary forecasts, reasonably accurate estimates might be forecasted based on administrative records from past years. Chapters 4–7 focus on the estimation of numbers of people eligible for WIC. Chapter 8 focuses on estimates of participation among eligible people and methods for estimating participation among them. In Chapter 9, we synthesize all our findings and provide different options for estimating eligibility and participation.
SUMMARY
This chapter began with an assessment of the accuracy of the USDA estimates to predict the number of WIC participants. Using available data to examine the accuracy of participation estimates, the panel found that the total number of participants matches the reported number quite closely. However, the numbers of participating infants and breastfeeding postpartum women are seriously underestimated, while the numbers of participating children and pregnant women are overestimated. The chapter also explains the two types of errors the panel investigated in its review of the current methodology—errors that cause systematic bias and prediction errors. Finally, in evaluating estimation methods, the panel considered accuracy, feasibility, characteristics of data sources, and correspondence of the method with current rules and their application at the local level.
Footnotes
- 1
Administrative records on the number of participants may include ineligible participants. Estimates of eligibility may differ from administrative records because of this, but we do not know how big these differences may be.