General Description

There are several steps that may be taken to identify the best predictors of inability to work (as indirectly measured by employment status and self-reported ability to work). To evaluate SSA's current Listings using the DMMS Wave 2 data, the following steps would be necessary:

1. Imputation of missing data.
2. Recoding of data as necessary for regression analysis.
3. Logistic regression of current Listings, with ability to work as the outcome measure and the items in the Listings as the predictor variables.
4. Receiver operating characteristic (ROC) analysis of diagnostic performance of the current Listings, using currently defined cut-points for distinction of positive and negative cases.

However, analysis of current Listings cannot be illustrated here because of two primary limitations. First, all patients in the DMMS Wave 2 study were on dialysis, and therefore it is not possible to enter whether a patient was receiving dialysis as a predictor variable. Second, individual physiologic symptoms as contained in the Listings are not measured the same way in the DMMS Wave 2 database. For example, the current Listings require persistent elevated serum creatinine for disability determination, whereas DMMS Wave 2 measured creatinine at a few distinct points of time.

Were we to find that the current Listings did not demonstrate acceptable prediction of inability to work, we would then perform a second series of analyses on all the relevant data in the DMMS Wave 2, using the following steps:

1. Imputation of missing data for relevant variables.
2. Evaluation of variable interaction.
3. Logistic regression analysis of "important" variables and interactions among variables.
4. Exploratory analysis of significant predictor variables.
5. ROC analysis of diagnostic efficacy of the resulting regression equation.

One could then determine the best combination of predictors of inability to work. Because employment status and self-reported inability to work are indirect measures of ability to work, such suggestions would rest on the assumption that these measures accurately represent inability to work. However, given that some patients who indicate that they are able to work are in fact not working (29 percent of those self-reportedly able to work full time), this may not be the case.

An illustrative analysis of the DMMS Wave 2 data for prediction of working status and self-reported inability to work 9 to 12 months after dialysis initiation is provided below. Although we were able to complete these analyses with the data available to us, the results should not be considered to be recommendations for changes in the Listings. Rather, at most, they may be used to guide future research.

Imputation of Missing Variables

One limitation of the DMMS Wave 2 was a large amount of missing data that made regression analyses unreliable. We initially performed sample regression analyses on randomly chosen halves of the database with death as the outcome variable. Probably because of erratically missing data for different patients in each set, the resulting regression equations for each half of the database did not contain the same predictor variables. This indicated a lack of reliability in the database, possibly because the regression process requires that all data be present for each patient.

There are, however, methods for making substitutions for missing data, referred to here as "imputation." Modern statistical computer packages (such as SPSS, used here) offer several different methods for missing variable imputation that are differentially appropriate depending on the characteristics of the variables. Listed below are some, but not all, of the options available:

• Series mean
• Mean of nearby points (specify # of points)
• Median of nearby points (specify #)
• Linear interpolation
• Linear trend at point

For the DMMS Wave 2 database, a normal distribution of values cannot be assumed due to the heterogeneous characteristics of the patient population. Patients with ESRD may be viewed as several diverse subgroups depending on the origin of the condition (e.g., young patients with glomerulonephritis, elderly patients with diabetes). Therefore, a series mean would not be appropriate; rather, the mean or median (depending on the continuous or categorical nature, respectively, of the variable in question) of nearby points (also called "next nearest neighbor") would be most statistically meaningful. The "nearby points" referred to in these methods are other cases (patients) who exhibit similar scores on other variables to the patient for whom the data point is missing. In most statistical packages, the analyst can specify the number of nearby points to be considered (two and up). For the purpose of our illustration here, we chose to use the mean or median of two nearby points -- the mean for continuous variables (such as most laboratory measures in this database) and median for categorical ones (such as presence of comorbidities, quality-of-life, and demographic measures).

Table E-1 indicates the number of available cases before and after data imputation for the variables entered into the regression analysis. The total number of patients was 546.

It is clear that SPSS did not necessarily impute data to fill in every missing point in the database and that the amount of imputation possible varied depending on the amount of data originally available. There are several variables here for which a majority of missing data has been imputed. This is considered "extrapolation" of data, and may not be reliable. We have therefore discarded data for which more than 30 percent of data are imputed (variables originally having data for about fewer than 400 patients). The loss of these variables from analysis limits the usefulness of the resulting diagnostic test because important variables may be missing.

Even once this imputation of data was completed, regression analysis on random halves of the database showed disparate results, indicating that the small number of patients in the database, in contrast to the large number of variables, continues to be a problem. For this reason, the results presented here should not be considered accurate, only illustrative.

After discarding extrapolated data and eliminating redundancies, there were 64 variables available for regression analysis.

Recoding of Variables

For both the purposes of assessing the current Listings and for the use of regression analysis, some recoding of the existing variables in the DMMS Wave 2 questionnaire was necessary (see Appendix A and Appendix C for existing coding). Note that we have not performed all of the necessary recoding for this illustration. Because PD and HD patients are equally represented in DMMS Wave 2, it would be necessary to either "weight" the HD cases so that they counted four times for every single PD patient, or to examine PD and HD patients separately. We have not done this here for simplicity's sake. We have, however, performed other required recoding, (e.g., we recoded multilevel categorical variables into several binary variables).

Table E- 1. Variables for which data were imputedAppendix E: Sample Analysis of DMMS Wave 2 Data

Table

Table E- 1. Variables for which data were imputedAppendix E: Sample Analysis of DMMS Wave 2 Data.

Investigation of Interacting Variables

There are many different methods for investigating possible interactions among independent variables, including a priori MANOVAs and neural networks. A full description of these methods is beyond the scope of this sample analysis.

This step was not taken for the purposes of this illustration; no interaction effects are entered into the regression equations below.

Logistic Regression Analysis

For the purposes of illustration, we have used the backwards stepwise entry method here. We have included a constant in the model due to the exploratory nature of the analysis. (The option is provided by SPSS to exclude the constant.)

The regression analysis results in an equation of the form:
Y = C + b₀x₀ + b₁x₁ + b₂x₂...b_nx_n
where Y is the outcome variable (here, full time employment coded as 0 or 1), b_0..n are the coefficients for each dependent measure, and x_0..n are the values of the variables included in the equation.

Analysis With Laboratory and Physiologic Measures

It was first important to assess prediction of ability to work using the variables of interest to SSA. Specifically, their Listings include physiologic and disease state measures to assess disability eligibility at step 3 of the disability sequential evaluation process. SSA does not evaluate other factors such as social, history, and demographic issues at this step of the process.

We therefore used only the 40 laboratory and physiological measurements listed in the above table for prediction of employment status and self-reported ability to work:

1. Categorical Variables: 1 = presence of condition; 0 = Absence:
   Treatment modality Ethnicity Angina CABG Cardiac arrest Cerebrovascular disease TIA PVD Absent foot pulse Claudication Congestive heart failure Pericarditis Pulmonary edema Lung disease Neoplasm Diabetes Hypertension Glomerulonephritis Smoking status
2. Serum Ca Phosphorus Serum icarbonate Hematocrit Hemoglobin Serum creatinine before first dialysis Serum creatinine at first dialysis BUN before first dialysis Predialysis BUN Postdialysis BUN Predialysis weight Postdialysis weight Serum aluminum Cholesterol Triglycerides Serum intact PTH Age Median predialysis SBP Median predialysis DBP Dry weight BMI Median predialysis weight

It must be remembered, when reading the results below, that the outcome measures are surrogates of true ability to work. Therefore, these equations cannot be considered representative of results one might actually use in a disability evaluation process. This is merely an illustrative example.

Analysis With Laboratory and Demographic Measures

Illustrated below are the regression analyses performed using demographic and laboratory values for prediction of the outcome variables.

Sixty-four variables were entered into the equation at Step 1:

1. Categorical Variables: 1 = presence of condition/characteristic 0 = Absence:
   Diabetes Hypertension Glomerulonephritis Single Married Widowed Divorced Full time Retired Disabled Clerical Professional Tradesperson Manual labor Student White Black Other race CHD Full time 2 yrs previous Part time 2 yrs previous Working part time at start of dialysis Education less than high school High school graduate College graduate Treatment modality Ethnicity Angina CABG Cardiac arrest Cerebrovascular disease TIA PVD Absent foot pulse Claudication Congestive heart failure Pericarditis Pulmonary edema Lung disease Neoplasm Independent eating Independent transferring Independent ambulating
2. Continuous variables:
   Serum Ca Phosphorus Serum bicarbonate Hematocrit Hemoglobin Serum creatinine before first dialysis Serum creatinine at first dialysis BUN before first dialysis Predialysis BUN Postdialysis BUN Predialysis weight Postdialysis weight Serum aluminum Cholesterol Triglycerides Serum intact PTH Age Median predialysis SBP Median predialysis DBP Dry weight BMI Median predialysis weight

Summary

These illustrative regression analyses have shown that laboratory and physiological measures alone do not take into account all the factors that lead to an individual's decision whether he or she does or can work. Given the current sociological and demographic issues that confront a patient on dialysis, it may be reasonable to suggest that physiologic values alone cannot predict working status. The equations that resulted from the inclusion of both physiologic and sociodemographic variables fared much better and were able to account for about half of the variance in the outcome variables. However, because these results were not able to be replicated on random halves of the database, none of the findings can be considered definitive.

However, these results cannot be considered indicative of what would be appropriate for disability evaluation. Because the outcome measures were surrogates of ability to work that we know are not accurate substitutions, the results may be quite different if a more direct measurement were used as the outcome measure.

If these equations could be considered useful and accurate, additional steps would be necessary to identify the best predictors of ability to work. First, these equations treat continuous variables without considering any predefined "cut point" for diagnosis of positive versus negative. For example, in the current Listings, a cut point of 4 mg/dL is used for serum creatinine, above which the individual is considered disabled, and below which the individual is considered able to work (in combination with other factors). Our above regression analyses treated continuous variables as continuous, rather than redefining them as binary (positive/negative) with a cut point.

Once the regression analysis has identified key predictor variables, one may then want to do some exploratory analyses to see if even better results could be obtained were the continuous variables translated into binary ones using a diagnostic cut point. It may be useful to do an ROC analysis, like the one illustrated below, for individual predictor variables to identify the most appropriate cut point.

Second, reporting the amount of variance accounted for by these equations does not portray an accurate picture of the usefulness of these equations in a diagnostic process, because the statistic does not reflect the type of errors that are occurring when the equation is used. If the equation was resulting in too many people being considered "disabled," this may be an acceptable type of error. However, if too many people were being considered able to work, then the use of this diagnostic equation may lead to denial of disability insurance to individuals who need it. The sample ROC analysis, below, offers methods for assessing the practical accuracy of such combinations of predictor variables.

ROC Analysis

In order to best assess an equation's diagnostic capabilities, it is necessary to examine its diagnostic test characteristics in the form of ratios expressing the percentage of cases correctly classified. As we mentioned in the Descriptive Statistics section of this report in our univariate example of diabetes as a predictor of employment, diagnostic test characteristics are essential because significance tests do not accurately depict the amount of diagnostic error a test produces.

There are four types of test characteristics commonly considered:

• True positive (TP) (patient has condition, test detects condition)
• False negative (FN) (patient has condition, test fails to detect it)
• False positive (FP) (patient is normal, test mistakenly detects condition)
• True negative (TN) (patient is normal, test finds patient normal)

Appendix E: Sample Analysis of DMMS Wave 2 Data

Table

Appendix E: Sample Analysis of DMMS Wave 2 Data.

Sensitivity = TP / (TP + FN) [the proportion of patients with the disease who are detected by the test]

Specificity = TN / (TN + FP) [the proportion of patients without the disease who are correctly diagnosed as negative]

Positive predictive value (PPV) = TP / (TP + FP) [the proportion diagnosed positive that are truly positive]

Negative predictive value (NPV) = TN / (TN + FN) [the proportion diagnosed negative that are truly negative]

In the cases of the above equations, the "positive" case is when Y = 0, or the patient is not working full time (or reporting that he or she is not able to work full time). Conversely, when Y = 1, the case is considered "negative," and thus the person is self-reportedly able to work or working full time. Thus, a true positive for this equation is when the equation predicts that a patient will not be working, and indeed, the patient is not working.

It is also important to note that in the examples provided here, a "false positive" may not truly be "false" because no "gold standard" is available to determine whether the equation is right or wrong, versus whether the surrogate outcome measure is right or wrong at portraying ability to work. The test characteristics displayed below really reflect the predictive value for working status or self-reported ability to work, not true ability to work.

In the above equations, rarely does Y equal exactly 1.0 or 0.0. For each patient, the predicted value of Y ranges from 0 to 1. The equation will not work well for some patients, who may score around 0.5. It is then important to determine whether such patients are considered "positive" or "negative" when using this test. This is called the "cut value" or "threshold." The diagnostic test characteristics are also influenced by where the cut value is placed. The default value in SPSS is 0.5. The threshold can be varied from 0 to 1 and graphed in a ROC curve, as will be shown below.

The following table indicates the test characteristics for the equation above predicting working status using only physiologic variables when the cut value is 0.5:

Appendix E: Sample Analysis of DMMS Wave 2 Data

Table

Appendix E: Sample Analysis of DMMS Wave 2 Data.

Notice in this table that most of the errors result from the false-positive designations-individuals predicted by the equation as not working who are in fact working. If this equation were used in the disability evaluation process, however, this error would likely have no effect; these individuals are working and are therefore not applying for disability. The only truly effectual error is that in which patients who may be disabled are denied disability. This may happen with the false-negative cases, 14 individuals who were predicted to be working who are in fact not working¹. This is 3.6 percent of the entire population who, if this were prediction of ability to work, might be receiving disability payments while no longer considered disabled.

These statistics suggest that an equation with just physiologic variables may be much more useful than indicated by the regression analysis described above. However, this assumes that working status is a true measure of ability to work, which is likely not the case.

The following table provides information on the equation that used both physiologic and sociodemographic variables when the cut value is 0.5:

Appendix E: Sample Analysis of DMMS Wave 2 Data

Table

Appendix E: Sample Analysis of DMMS Wave 2 Data.

This table shows that the inclusion of sociodemographic variables improves the specificity substantially, and thus more individuals truly able to work are classified correctly as such. The improvement comes as a result of a lower false-positive rate; few patients who are working full time would be classified as disabled. The improvement yielded by this equation, therefore, does not lead to any real improvement in disability evaluation, because these 25² "false-positive" individuals would be working and therefore not apply for disability. Only 6.9 percent (26 out of 366) of the population considered here that might be receiving disability would be moved off disability rolls (if the outcome measure were an accurate representation of ability to work). This statistic is higher than that above, when only physiologic variables are used (3.6 percent).

Assuming that the surrogate outcome measure of working status could be used as a gold standard for ability to work (which, as discussed above, it cannot), an ROC curve could then be generated that represented how the test characteristics of these equations change with different cut points. The results could then be used to choose a cut point that would best fit the needs of the diagnostic test. Most diagnostic tests prefer high sensitivity and low specificity versus the opposite, so that no positive cases are missed. Other diagnostic tests work best if a balance is reached between sensitivity and specificity. In this case, disability evaluation may work best if sensitivity is maximized.

Figures E-1 and E-2 show the regression equations for predicting working status in ROC space, where sensitivity is plotted against 1-specificity. E-1 includes only physiologic variables, while E-2 also includes sociodemographic variables. In this space, a curve that falls closer to the upper left quadrant indicates better prediction value (higher sensitivity and specificity). This curve shows that as sensitivity increases specificity decreases, and vice versa. It is therefore possible to choose a cut point/threshold that will maximize either sensitivity at the expense of specificity, or vice versa, or to balance the two out, as is done with a cut point (Y-value) of 0.5, marked on the graph. When sensitivity is maximized at the expense of specificity (as might be done with a cut point of 0.7, shown on the graphs in Figures E-1 and E-2), most patients will be assigned to the category of "Not working full time," and many of these will be erroneous assignments. Because of the small proportion of patients who are working full time at followup, this may not be considered a major error.

Figure

Figure E- 1. ROC curve of physiologic variables for prediction of working status at 9- to 12-month followup.

Figure

Figure E-2. ROC curve of demographic and physiologic variables for prediction of working status at 9- to 12-month followup.

On the other hand, if specificity were maximized at the expense of sensitivity (as might be done with a cut point of 0.3, shown on the graph below), more patients would be identified as being able to work; but some of these would be individuals who are not able to work and would be erroneously denied disability.

Summary

These sample analyses have illustrated the statistical methods that may be used to determine predictors of disability when data such as that in DMMS Wave 2 are available. The results presented here cannot be considered indicative of the results that might be found if an unflawed measure of ability to work were available to use as the outcome measure.

We have illustrated the importance of having available data for all cases to be included in the regression analysis and outlined ways of identifying the best variables to enter into the regression equation. The results of the regression equation, however, should be considered a starting point for identifying the best combination of predictors and should not be considered the final solution. In particular, the results provided by regression analysis do not indicate the diagnostic value of the equations and, thus, ROC analysis should accompany the results. Additional exploratory analysis of significant predictor variables would then be prudent.

Footnotes

1

The fact that so few individuals were working also suggests that we have overfit some of the multiple regression equations. Overfitting is particularly likely when an equation consisting of many variables is based on only a few events.

2

As in the preceding example, this is a relatively small number of events; therefore,we may have overfit the data.