Data:

We extracted Medicare data from 2013 and 2014 related to hospital readmission for 3 conditions: chronic obstructive pulmonary disease (COPD), congestive heart failure (CHF), and total hip/knee arthroplasty (THA/TKA). For each condition, we extracted cases (patients readmitted within 30 days of hospital discharge) and controls (patients not readmitted within 90 days of discharge, matched by age, sex, and race). The data were randomly divided into a training data set and a replication data set consisting of case-control matched pairs. Variables included 42 unique comorbidities highly prevalent in older adult patients and were extracted from the Elixhauser, Charlson, and Centers for Medicare & Medicaid Services condition category indices, in addition to condition-specific variables identified by the stakeholders. Comorbidities within each condition that were significant in both the training and replication data sets were selected for further analysis.

Analysis:

We analyzed the data using 3 modeling methods:

1. Visual analytical modeling was conducted by (a) representing the patients (only cases) and their comorbidities as a bipartite network; (b) identifying patient subgroups and their most frequently co-occurring comorbidities through the use of bicluster modularity maximization, and testing the statistical significance of its degree of clusteredness through comparisons to 1000 random permutations of the data; and (c) comparing the comorbidity co-occurrence between the training and replication data sets through the use of the Rand Index and testing its significance. The stakeholders examined the bipartite networks of readmitted patients and their comorbidities to assess their clinical significance and to design potential interventions for reducing the risk of readmission.
2. Classification modeling was conducted by (a) building and validating a multinomial logistic regression model to predict the probability of a patient (cases and controls) belonging to each of the subgroups identified by the visual analytics based on their profile of the comorbidities; and (b) classifying each patient (cases and controls) to the subgroup to which the patient had the highest probability of belonging, which was used in the subsequent predictive modeling.
3. Predictive modeling was conducted by (a) building and validating a logistic regression model using data from all the patients lumped together to predict the probability of a patient being readmitted to the hospital, based on the patient's profile of comorbidities, age, sex, and race; and (b) using these variables and standard predictive modeling discrimination measures (concordance statistic [C statistic], calibration plots, net reclassification improvement [NRI], and integrated discrimination improvement [IDI]) to compare this lumped model to the corresponding hierarchical logistic regression models containing information about the patient subgroups (identified from the visual analytics).

Data limitations:

Because the baseline models were built using Medicare administrative data, we used the same data for our models to enable a head-to-head comparison. However, although such administrative data have strengths, including being representative of the population aged >65 years in the United States and having multiple years enabling replication analysis, they have known limitations, such as the lack of comorbidity severity and test results, which could strongly impact the accuracy of predictive models. Research using electronic medical records containing a wider range and more granularity of variables, such as laboratory results and medications, could help overcome such limitations. Furthermore, comorbidities on their own do not have a strong enough signal for readmission, and future approaches should explore other variables, including medications, to predict readmission.

Method limitations:

Our development and use of visual analytics for building stratified regression models revealed 2 limitations related to analyzing large data sets:

1. Although our overall method scaled well to analyze data sets ranging up to 25 000 patients, the measurement of significance for biclustering and replication took up to a week of computation despite using a dedicated server with multiple cores. This high computational cost was caused because significance of clustering is currently measured by comparing the clustering results to 1000 random permutations of the data. Future research should explore approaches to measure the significance of modularity using mathematical approaches (through formulas) rather than permutation approaches.
2. Although bicluster modularity was successful in finding significant and meaningful biclusters, the visualizations were extremely dense and therefore concealed patterns within and between biclusters. Future research should explore more advanced methods to enable interpretability of results within and between large and dense biclusters. Finally, following recommendations from the stakeholders, we pooled 2 years of the data and randomly split them to form the training and internal validation data sets. Future research should therefore externally validate the results using a different year from the Medicare database.

Heterogeneity in Humans

A wide range of studies^1-9 on topics ranging from molecular to environmental determinants of health have shown that most humans tend to share key characteristics (eg, comorbidities or genes) that form distinct patient subgroups. A primary goal of precision medicine is to identify such patient subgroups and infer their underlying disease processes to design interventions that are targeted to those processes.^2,10 For example, recent studies in complex diseases such as breast cancer^3,4 and asthma^5-7 have revealed molecular-based subphenotypes, each with different underlying biological mechanisms precipitating the disease, and therefore each requiring different treatments. Because such targeted treatments can have a profound impact on patient outcomes, the identification and comprehension of patient subgroups is critical to patient-centered outcomes research (PCOR).

Current Approaches for Identifying Patient Subgroups

A patient subgroup is defined as a subset of patients drawn from a population (eg, older adult patients) that share ≥1 characteristics (eg, renal failure and diabetes). Patients have been divided into subgroups by using (1) investigator-selected variables, such as race, for developing stratified regression models¹¹ or assigning patients to different arms of a clinical trial; (2) existing classification systems, such as the Medicare Severity Diagnosis Related Group,¹² to assign patients to a disease category for billing purposes; or (3) computational methods, such as classification^13-15 and clustering,^5,16 to discover patient subgroups from data.

One of the simplest unsupervised computational methods for identifying patient subgroups is by enumerating conjunctions of variables, such as by analyzing all dyads and triads of co-occurring comorbidities in the Medicare database,¹⁷ and then examining the most prevalent subgroups. Other methods attempt to partition a data set of patients and characteristics into sets that are relatively homogeneous. These sets can either be 1-sided clusters (clusters of patients, or clusters of characteristics) or biclusters^16,18,19 (clusters of patients and characteristics). K-means and hierarchical clustering^15,16 are among the most commonly used 1-sided clustering methods and require inputs such as a similarity measure (eg, Jaccard similarity) and the expected number of subgroups, but with no agreed-upon approaches to automatically determine them. More recently, biclustering^16,18,19 methods (also called co-clustering methods) have been developed to automatically identify nonoverlapping or overlapping submatrices consisting of both patients and characteristics. Compared with partitioning methods that use similarity measures to identify clusters, dimensionality reduction methods are used to attempt to find a reduced dimensional space where differences among patients are maximized. For example, principal component analysis¹⁵ (PCA) is used to identify principal components, which are weighted combinations of characteristics along which patients have the maximum variance. The patients' scores are projected onto a plane typically defined by the 2 most important principal components. Methods such as k-means are then used to identify clusters of patients in this reduced dimensional space.

In contrast to the above-described unsupervised methods, supervised methods focus on identifying patient subgroups by taking into consideration outcome variables (eg, responders and nonresponders in a treatment arm). For example, classification and regression trees (CARTs),¹⁵ as well as enhancements such as random forests²⁰ and bump hunting,¹⁴ progressively divide patients into subgroups based on the outcome variable by using conjunctions of patient characteristics at each step. The method outputs a tree, and each path from the root node to a leaf node defines a patient subgroup.

Bipartite Network Analysis

This project focuses on using a powerful unsupervised visual analytical method called bipartite network analysis²¹ to automatically identify and visualize patient subgroups based on multiple characteristics. Visual analytics is defined as the science of analytical reasoning facilitated by interactive visual interfaces.²² Bipartite network analysis is an unsupervised visual analytics²² method used to help discover complex patterns in data. Bipartite networks have been used extensively to analyze a wide range of associations, including social networks,²³ disease classification,²⁴ and drug discovery,²⁵ and to identify proteomic heterogeneity.²⁶ The principal investigator (PI) has developed a general methodology to use bipartite networks to identify patient subgroups based on their molecular profile²⁶ and applied it in a number of diseases, including asthma,²⁷ rickettsial diseases,²⁸ and influenza.²⁹ This methodology enabled the identification of a new classification of asthma based on complex inter-subgroup associations and their underlying mechanisms.²⁷

A network consists of a set of nodes (represented as shapes such as circles) connected in pairs by edges (represented as lines).²¹ Unipartite networks (Figure 1A), commonly used to model social networks, have nodes that represent 1 type of entity (eg, patients), while the edges represent associations among them (eg, friendship links). In contrast, bipartite networks (Figure 1B) represent 2 types of entities (eg, patients and characteristics), and edges exist only between different types of entities representing a specific relationship (eg, a patient has diabetes).²¹

Figure 1

Comparison Between a Unipartite (A) and a Bipartite Network Representation (B), and How a Bipartite Network Can Reveal Biclusters Containing Patient Subgroups and Their Most Frequently Co-occurring Comorbidities (C).

Based on a wide range of projects that the PI has conducted to identify patient subgroups in different diseases,^26,27,30-35 in addition to a preliminary analysis of readmission data discussed later in this report, bipartite networks appear to be powerful for identifying patient subgroups and their underlying mechanisms based on 2 fundamental properties:

• Automatic identification of the number, size, and significance of patient subgroups. A network is a graph with mathematical properties,²¹ and the field of network analysis has developed a large collection of quantitative measures to analyze complex associations in data. For example, modularity algorithms can automatically (ie, requiring no user input) identify in large networks the number of clusters, their members, the degree of clustering with respect to random networks of the same size,^21,36-38 and the significance of the clustering.^16,39 Modularity is defined as the fraction of edges falling within a cluster (ie, the expected fraction of such edges in a network of the same size with randomly assigned edges).²¹ Modularity ranges from −0.5 to +1. Modularity algorithms search for a partition of nodes that optimizes the modularity quantity. This approach has been developed for identifying uniclusters in unipartite networks and biclusters in bipartite networks.⁴⁰ For example, we have used modularity to identify biclusters of patients and their characteristics in readmission data.^41-43 Although there is active research in developing extensions to the modularity algorithm, its basic approach is a well-accepted and automatic method for finding clusters in complex networks.²¹
• Visual representation enabling comprehension of quantitative results. Although modularity algorithms identify the number and membership of biclusters, they cannot reveal how individual nodes or subgroups are related to each other, which has considerable explanatory power to aid comprehension of patient subgroups. To enable comprehension of these relationships, networks are typically laid out using force-directed algorithms that pull together nodes that are strongly connected and push apart nodes that are not. The result is that nodes with a similar pattern of connections are placed close to each other, and those that are dissimilar are pushed apart. As shown in Figure 1C, the application of the Kamada–Kawai (KK) force-directed algorithm⁴⁴ to a bipartite network has revealed 2 biclusters, each consisting of patients and comorbidities, 1 on the left and 1 on the right. The patients in each bicluster are defined as a patient subgroup, and the comorbidities in each bicluster are the most frequently co-occurring comorbidities for that patient subgroup. Because such layouts are approximate, the patient subgroups identified from a modularity algorithm are superimposed on the layout (as shown by the ovals in Figure 1C).

The resulting bipartite visual representation reveals 2 important relationships enabling comprehension of subgroups:

• Relationships within patient subgroups include the number and proportion of patients and characteristics in each cluster (eg, a bicluster containing few patients and many variables and another containing many patients with few variables) or patient subgroups that have high or low uniformity (eg, the left- and right-side patient subgroups in Figure 1C).
• Relationships across patient subgroups include visualizing the proportion of additional characteristics across biclusters. For example, by coloring nodes in a network by race or sex, the visualization could reveal differences according to these characteristics across the subgroups (eg, an Asian-dominated cluster that has chronic obstructive pulmonary disease [COPD] and congestive heart failure [CHF] vs a Hispanic-dominated cluster that has osteoporosis and renal failure). Finally, the visualization also can reveal intercluster associations (eg, the intersubgroup associations shown in Figure 1C).

Although each of the above-described relationships can be calculated without the visualization, the simultaneous presentation of them in the bipartite network visualization leverages the parallel processing power of the visual cortex of the human brain,⁴⁵ enabling the discovery and comprehension of intra- and intercluster associations. We have demonstrated^26,27,30-35 that this integration of associations, once tested for significance, can enable stakeholders to infer the underlying processes in each patient subgroup and thus design targeted interventions. Furthermore, as shown by the edges that cross the biclusters, patients in 1 bicluster can have characteristics present in another bicluster, albeit far less frequently compared with the comorbidities in their own biclusters. Therefore, based on the 2 biclusters in Figure 1C, a patient with C1 is also highly likely to have C3 and C5, but less likely to have C2 or C4, and vice versa. Consequently, patient subgroups tend to have a similar but not necessarily identical profile of comorbidities. Membership of a new patient in a bicluster is therefore probabilistic, based on the entire comorbidity profile, and therefore reflects the true variation of comorbidities in humans. This probabilistic approach is different from the approach of defining patient subgroups based on a few inclusion or exclusion criteria (eg, CHF and diabetes) while ignoring the rest of the comorbidity profile.

Stratified Regression Modeling

Although numerous studies have revealed considerable heterogeneity across patients, automatically identified subgroups have not been systematically used to develop predictive models. Consider the following logistic regression model with the 3 predictors age, sex, and body mass index (BMI) used to predict the risk of type 2 diabetes, simplified for illustration:

This model can predict, for example, that patients John, Mary, and Tom, each with different profiles of age, sex, and BMI, have different risks for developing type 2 diabetes of 0.9, 0.6, and 0.3, respectively. Although the model shown above uses the characteristics of each patient to estimate their individual risks, each estimate is derived using the same coefficients (β₁, β₂, and β₃) for the same set of variables (age, sex, BMI). Therefore, the underlying assumption is that a single model can accurately estimate the risk for all patients in a cohort.

However, as stated earlier, a “one size fits all” assumption is not well aligned with the notion of heterogeneity in patient populations that is observed in numerous diseases and conditions.^1-9 For example, the sample regression model discussed above ignores the possibility that John and Mary might be similar because they share the same processes resulting in diabetes, whereas Tom might have an entirely different process leading to the same disease. In such situations, a single model might not be as accurate as having 2 models, each targeting different patient subgroups formed by similar profiles—an approach referred to as stratified regression modeling.

The mathematical intuition for using stratified regression models is that they can achieve better fit to subsets of the data that are homogeneous, compared with a single regression model that is fitted to all the data,¹¹ an approach well established in biostatistics. However, as stated before, stratification is typically done using the conjunction of a few investigator-selected variables. Although this approach can be successful sometimes, it does not consider multiple characteristics in a data set. In contrast, bipartite network analysis provides the opportunity to automatically identify patient subgroups based on many characteristics, refine them through stakeholder engagement, and then incorporate them in the design of stratified regression models. Therefore, if the subgroups identified are indeed strongly and significantly clustered, they should improve the accuracy of predictive models that do not exploit such patient subgroups.

The High Cost of Hospital Readmission

Although our approach to the automatic identification of patient subgroups and its use in stratified regression modeling is generally applicable to analyzing a wide range of patient-characteristic data sets, there is an urgent need⁴⁶ to apply it to address the hospital readmission problem in older adult patients. An estimated 1 in 5 older adult patients (>2.3 million Americans) is readmitted to a hospital within 30 days after being discharged.⁴⁶ Although many readmissions are unavoidable, an estimated 75% of readmissions are unplanned, and mostly preventable.⁴⁷ Unplanned hospital readmissions nationwide therefore impose a significant burden in terms of mortality, morbidity, and resource consumption. For example, a hospital readmission of a patient with a hip fracture can easily negate the functional gains painstakingly achieved through weeks of postacute rehabilitation. This loss is over and above the costs to caregivers and relatives, who have to relive the stress of the original hip fracture episode, reorganize their work schedules to care for the patient, and restart the rehabilitation process after discharge. Across all conditions, unplanned readmissions cost almost $17 billion annually in the United States,⁴⁷ making them an ineffective use of costly resources in addition to being closely scrutinized as a marker for poor quality of care by organizations such as the Centers for Medicare & Medicaid Services (CMS).⁴⁸

Current Risk Prediction Models for Readmission

Several national and local efforts have developed models to predict the patient-specific risk of readmission. However, there is considerable room for improvement in their predictive power. For example, researchers at Yale University developed a model to predict the readmission of patients with CHF, based on patient profiles such as patient age, sex, and comorbidities.⁴⁹ Similar models have been developed for predicting the readmission risk of patients admitted for COPD⁵⁰ and total hip/knee arthroplasty (THA/TKA).⁵¹ However, because the predictive power (measured by the concordance statistic [C statistic]) of these models is in the range of 0.60 to 0.65, there is considerable room for improvement. We have achieved some improvement in predictive power for COPD (C statistic = 0.72) by introducing factors related to postdischarge management (eg, antibiotic use and follow-up after discharge).⁵² However, an accurate predictive model based solely on a patient's health profile at or during hospitalization will be more helpful for planning targeted postdischarge care. In this project, we used discrimination (ie, C statistic, discrimination slope), calibration (ie, calibration-in-the-large, calibration slope), and reclassification (ie, net reclassification improvement [NRI], integrated discrimination improvement [IDI]) measures discussed in aim 2 to test whether information about patient subgroups identified through visual analytics will improve the accuracy of prediction when compared with existing models that do not consider information about patient subgroups.

Importance of Comorbidities in Predicting the Risk of Readmission

Studies have shown that the number and type of comorbidities play a critical role in determining the risk of readmission.⁴⁶ For example, almost two-thirds of older adult patients population have ≥2 comorbid conditions, resulting in a heightened risk of adverse health outcomes, such as hospital readmission.¹⁷ Furthermore, multiple comorbidities very often do not act independently but rather interact with each other, resulting in processes that can precipitate readmission.⁵³ For example, because of the systemic nature of renal disease, a patient with hip fracture, CHF, and renal failure is at a higher risk of renal failure exacerbation resulting in a readmission than is a patient who only has renal failure.^41,42 Additional evidence for different comorbidities precipitating readmission came from discussions with 3 readmitted patients (see Patient and Stakeholder Engagement section). The aforementioned studies and our discussions with readmitted patients suggest that comorbidity profiles can cause considerable heterogeneity among patients, each with different processes precipitating readmission and associated risks.

Data Source

We used 100% Medicare claims data for years 2011-2014 because (1) the scale of the data enables subgroup identification with sufficient statistical power; (2) the spread of data collected from across the United States enables generalizability of the results; (3) the population consisting of older adult patients enables research on an underrepresented segment of the US population; (4) the variables include comorbidities critical in readmission and demographics enabling analysis of differences by race and sex; and (5) the use of the data to build readmission predictive models enables head-to-head comparison with the subgroup-specific modeling we will develop. (Limitations of Medicare data are discussed in the “Conclusions” section.)

Study Population

We analyzed patients hospitalized for COPD, CHF, and THA/TKA. We selected these 3 conditions because (1) hospitalizations for each of these conditions are highly prevalent in older adult patients,⁴⁶ (2) hospitals report very high variations in their readmission rates,⁴⁶ and (3) for each of these conditions there exist well-tested readmission prediction models that did not consider patient subgroups.^49-52,55 For example, as shown in the regression equation shown earlier, although these models include comorbidities and demographics as predictors, they do not take into consideration the existence of patient subgroups that share many comorbidities. For each index condition, we used 100% Medicare claims data on readmitted patients in 2013 and 2014, removed patients who had duplicate records, and then randomly split the pooled data set to create the training and replication data sets. We extracted all patients who were admitted to an acute care hospital on or after July 1, 2013, had a principal diagnosis of the index condition, were aged ≥66 years, and were enrolled in both Medicare parts A and B fee-for-service plans. Because payment to primary care physicians participating in Medicare Advantage (MA) plans is capitated, and the physicians are not required to file claims for MA patients, these patients do not have complete data (eg, comorbidities) relevant to our study. Therefore, we excluded patients with MA coverage from 6 months before to 1 month after the discharge of the index admission. To determine how this exclusion affects the generalizability of our results, we compared the demographics of patients with and without MA coverage and here report whether our results generalize to MA patients. Because the published CMS models did not use Medicare Part D related to prescription medications, we did not include those data in our analyses.

After extracting the cases (defined as patients readmitted within 30 days of hospital discharge), we extracted an equal number of controls (defined as patients not readmitted within 90 days of discharge, matched by age, sex, race, and Medicaid eligibility as a proxy for economic status). We performed individual matching by randomly selecting a control to match a case on age, sex, race/ethnicity, and Medicaid eligibility.⁵⁶ The 90-day window of no readmittance represents an episode of care proposed by CMS for patients,^57,58 indicating that the controls are substantially free of complications that result in readmission during this period, and therefore allowing an effective comparison with the cases. This method of defining controls was identical to that used by the CMS models, which we therefore had to use for a fair comparison. We used this method for both the training data set and the replication data set. A small percentage (0.8%) of Medicare patients had “unknown race” recorded for the race attribute. Therefore, we had an equal number of patients of unknown race in cases and controls.

To match the exclusion criteria of the published CMS models, we excluded patients who were transferred from other facilities, died during the readmission hospitalization, or transferred to another acute care hospital. For this aim, we analyzed only data from patients who were readmitted within 30 days after discharge. Finally, we used the union of comorbidities in the 3 comorbidity indices described earlier: Elixhauser and Charlson comorbidity indices, and CMS CCs. To guard against miscoding of comorbidities, we used a 6-month look-back period in addition to the year the data were collected to ensure all comorbidities of the patients were included. Appendix A shows the detailed inclusion and exclusion criteria used to extract cases and controls for COPD, CHF, and THA/TKA; the respective numbers of patients extracted at each step; and the ICD codes for each of the 3 index conditions selected for analysis.

Measures

The outcome of interest was whether a patient with the index admission had an unplanned readmission to an acute care hospital in 30 days after discharge, based on data extracted from the MEDPAR file. Predictors for the risk of readmission model for each disease condition were the same patient demographics (eg, age and sex) and condition-specific variables (eg, oxygen dependency for COPD) as in the existing predictive models, in addition to the comorbidities we describe next.

Established comorbidity indices were used to form the candidate predictor pool, including the Charlson and Elixhauser comorbidity indices and the CMS CCs in the condition-and procedure-specific readmission models.⁵⁹ All these indices have mapped ICD-9-CM diagnosis codes to identify selected comorbid conditions. The Charlson Comorbidity Index is calculated as a weighted score from 17 conditions.⁶⁰ Elixhauser methods identify 31 conditions often used individually in regression models for prediction or risk adjustment.⁶¹ The CMS CCs are used for risk adjustment for various outcomes, such as readmission, mortality, and complication. We extracted the conditions included in these indices using corresponding diagnostic and procedure codes from physician and outpatient and inpatient claims in the 6 months before or on the day of the index admission. Because these comorbidity indices contain common as well as unique conditions, a union of the indices was prepared for analysis. This list of comorbidities was presented to the stakeholders, who suggested additional variables for each index condition that were important for analyzing hospital readmission; this resulted in the following variables in each condition: for COPD, history of sleep apnea and mechanical ventilation; for CHF, history of coronary artery bypass graft surgery; and for THA/TKA, congenital deformity of the hip joint and posttraumatic osteoarthritis.

Feature Selection

The visual analytical modeling was conducted with comorbidities, whereas the subsequent predictive modeling was done using the same variables that were included in the existing Yale models with which we were conducting our comparison. We used the following method to select features to include in our visual analytical modeling: (1) We removed comorbidities with prevalence <1%. This approach is typically used in other studies,⁶² because low-prevalence comorbidities tend to be unreliable, representing noise in the data. (2) We selected significant comorbidities in the training data set based on 2-way interaction tests using odds ratios with directionality and corrected for multiple testing using the Bonferroni method. (3) The remaining comorbidities were tested for replication in the replication data set. Appendix B shows the number of comorbidities and overall variables that were included in the analysis for each of the 3 conditions.

Visual Analytical Modeling

We developed a visual analytical method to identify patient subgroups in each of the 3 index conditions. The visual analytical method was developed using R scripts and applied using the following steps:

• Represented data as a bipartite network. As shown in Figure 1C, we used a bipartite network to model the cases (30-day readmitted patients) and their comorbidities (features selected above) with each condition. The nodes represented the cases and comorbidities, and edges represented which patient had which comorbidity.
• Laid out the network using a force-directed algorithm. We used the KK algorithm,⁴⁴ a force-directed algorithm to lay out the bipartite network. This layout algorithm pulls together nodes that are strongly connected and pushes apart nodes that are not. The result is that nodes with a similar pattern of connections are placed close to each other, and those that are dissimilar are pushed apart.
• Identified biclusters and their degree of clusteredness, using a modularity algorithm. We used a modularity algorithm to identify the number of clusters, their boundaries, and quality of that cluster partitioning. Modularity is defined as the fraction of edges falling within a cluster, that is, the expected fraction of such edges in a network of the same size with randomly assigned edges.²¹ Modularity ranges from −0.5 to +1, with values >0.3 indicating substantial clustering.²¹ We used the bipartite version of modularity to find biclusters (consisting of patient and comorbidity nodes) in the network.
• Enhanced the network layout based on biclusters. Because the traditional KK algorithm often cannot fully separate clusters in large and dense networks, the network layout needs to be enhanced before it can be useful to the stakeholders for understanding the associations. Therefore, once the biclusters were identified through the bicluster modularity algorithm, we used the ExplodeLayout algorithm⁴³ to separate the clusters identified through bicluster modularity, with the goal of reducing the visual overlap among them and thereby enhancing their comprehensibility. This approach effectively preserved the distances of nodes within a cluster but not the distances of nodes across clusters.
• Evaluated the significance and internal validation of the readmitted patient subgroups. The patient subgroups identified through the modularity algorithm were tested for statistical significance and internal validation in the test data set. The significance of the biclustered modularity was measured by comparing it with a distribution of the same quantity generated from 1000 random permutations of the network by preserving the network size (number of nodes) and the network density (number of edges). Next, we tested the replicability of comorbidity co-occurrence in the test data set using the Rand index.⁶³ This index measures the proportion of comorbidity pairs that co-occurred and did not co-occur in a cluster in both years (where 0 = no internetwork cluster similarity, and 1 = total internetwork cluster similarity). Finally, we tested the significance of the Rand index by comparing it with a distribution of the same measure generated from 1000 random permutations of the 2010 and 2009 networks. All tests of statistical significance were 2 sided.

Classification Modeling

As shown in Figure 1, the biclusters identified through the modularity algorithm contain patient subgroups and their most frequently co-occurring comorbidities with respect to other patients in the network. However, as shown in both figures, many edges go across biclusters, which demonstrates that many patients within a bicluster have comorbidities that exist in other biclusters. As is true for all partitioning clustering methods, including modularity, membership of a new patient in each bicluster is therefore probabilitistic. This means that classification of a patient into a cluster is not defined by the inclusion or exclusion of comorbidities (eg, CHF and diabetes) but rather by the probability of being in a patient subgroup. This reflects the true variation in comorbidities where patients are similar or different, not just in a handful of carefully selected comorbidities while ignoring others; rather, patients are similar or different based on all of their considered comorbidities. It is this overall profile of patients that reflects the reality of comorbid conditions and therefore is most likely to reveal the real-world processes precipitating readmission.

Therefore, to determine whether a specific patient belongs to a patient subgroup, we developed a classifier that provided the probability of a patient belonging to each subgroup. Such probabilities provide information such as whether a patient has a much higher probability of being in 1 patient subgroup compared with others or whether the patient has an approximately equal probability of belonging to 2 patient sugroups, in which case the patient could have the probability of having multiple processes precipitating readmission.

To capture this complexity of how patients are similar or different based on their comorbidities, we developed the classifier using multinomial logistic regression.¹⁵ In contrast to support vector machines and decision trees, multinomial logistic regression has the advantage of generating “soft labels,” or probabilities for a patient to be in each patient subgroup, enabling an understanding of complex patterns in patients such as those who might belong equally to 2 different subgroups. To validate the classifier, we used a cross-validation approach, where 1000 random samples of 75% of the data were selected with replacement to build the prediction model. The classifier was then used to predict the subgroup membership for the remaining 25% of the patients as the basis of calculating the average accuracy across the 1000 cross-validated samples.

Predictive Modeling

To develop and evaluate subgroup-specific models, we used the same data sources and inclusion and exclusion criteria for all patients who were admitted for each condition, as described in the data selection section above. However, here we used the cohort consisting of the entire population in each condition. We classified all patients in each index condition into subgroups by applying the patient subgroup classifier developed in aim 1. The outcome measure was 30-day unplanned readmission (yes vs no) extracted from MEDPAR files (inpatient claims). We conducted the development and evaluation of subgroup-specific readmission prediction models through the following steps.

• Developed subgroup-specific models. Our goal was to develop predictive models with an optimal balance between predictive ability and complexity. Therefore, we used an iterative process consisting of modeling and feedback from the clinician stakeholders. The training set for model development was randomly selected from the patients who were admitted for the respective index condition. All models included age, sex, race, index condition severity (eg, use of mechanical ventilation and respirator dependence), and subgroup indicators. A small percentage (0.8%) of Medicare patients had “unknown race” (ie, the data for race were missing); therefore, we grouped “unknown race” and “other race” together in the analyses. Because only 0.8% of Medicare beneficiaries had missing data on race, the risk of bias was too low to warrant a sensitivity analysis.
• Using the subgroups identified in aim 1, we developed stratified logistic regression models for each subgroup. Each of these subgroup-specific models included the same variables selected through feature selection in an index condition and additional ones deemed to be relevant to hospital readmission by the stakeholders. Subgroup-stratified modeling has the advantage of being more specific and interpretable but may not be the most efficient modeling approach. Therefore, we also incorporated information on patient subgroups into a single regression model (referred to as a hierarchical model) of all the data by inclusion of a subgroup variable. The model performance in the training sample was measured by goodness-of-fit statistics (eg, likelihood ratio test, deviance), C statistics, and Akaike information criterion (AIC). Given a set of models with satisfying goodness-of-fit, the preferred model should be the one with higher C statistic and lower AIC values. Coefficient estimates from the model were applied to the validation sets for model validation.
• Internally validated subgroup-specific models. We conducted internal validation to examine whether the prediction model had sustainable good performance in the validation sets. The model's performance in the validation sets was evaluated by the following measures:

  –

  Discrimination. This quantity measures the model's ability to distinguish patients with readmission from those without readmission. The C statistic is the most frequently used discrimination measure for logistic regression models and is identical to the area under the receiver operation curve. Model discrimination was also presented visually, using box plots showing the average risk prediction for patients with and without readmission.

  –

  Calibration. This quantity measures how well the predicted probabilities agree with observed probabilities. In logistic regression, calibration is measured by calibration indices (calibration-in-the-large and calibration slope) and visually illustrated by calibration plot (the scatterplot of the proportion of patients actually admitted vs deciles of predicted probability of having readmission). Good calibration is indicated by a calibration-in-the-large with a value close to 0 and a calibration slope close to 1. Because the study was overpowered from the large sample size, we did not measure the calibration based on statistical significance (eg, P values of the Hosmer-Lemeshow and calibration indices).

• Compare hierarchical models with the existing model that does not consider patient subgroups. The performance of our subgroup-specific prediction model derived from visual analytics was compared with the CMS models that include comorbidities and demographic variable as predictors but do not consider subgroups. Besides the comorbidities, all variables in the comparison sets were the same. We tested for discrimination and calibration using the following 2 measures:

  –

  NRI indicates the proportion of patients whose predicted probability of readmission improved (with reference to actual readmission status) between the CMS reference model and subgroup-specific model. We report 2 NRI statistics: (1) categorical NRI, which predicted readmission probabilities for each model divided into 10 sequential categories between 0 and 1, with improvement requiring a shift between categories; and (2) continuous NRI, which is based on the proportions of individuals with any improved predicted probability of readmission, regardless of the size of that improvement.

  –

  IDI indicates the difference in the average improvement in predicted risks between the CMS reference model and the subgroup-specific model.

Chronic Obstructive Pulmonary Disease

The application of the inclusion and exclusion selection criteria described in Appendix A resulted in a training data set (n = 14 508 matched case-control pairs, of which 51 pairs of patients with no comorbidities were dropped) and a replication data set (n = 14 508 matched case-control pairs, of which 51 pairs of patients with no comorbidities were dropped), matched by age, sex, race, and Medicaid eligibility (a proxy for economic status). The feature selection described in Appendix B resulted in 45 unique comorbidities identified from the 3 comorbidity indices (Elixhauser and Charlson indices and CMS CCs) plus 2 condition-specific comorbidities suggested by the stakeholders. Of these, 3 were removed because of <1% prevalence, and 30 survived the significance and replication testing with Bonferroni correction. These data were used for the subsequent visual analytical modeling

As shown in Figure 3, the bipartite network method identified 4 biclusters, each representing a subgroup of readmitted patients with COPD and their most frequently co-occurring comorbidities. The analysis revealed that the biclustering had significant modularity (Q = 0.17; z = 7.3; P = 2.813e−13) and significant co-occurrence replication (RI = 0.92; z = 11.62; P = 3.11e−31). Furthermore, as requested by the stakeholders, we superimposed the risk for each of the patient subgroups to enable stakeholders to determine whether the subgroups were clinically meaningful. Although the comorbidity co-occurrences within each bicluster were meaningful, because of the size of the data, the edges between clusters were too dense to be individually visible.

Figure 3

Results From the COPD Network Analysis Showing 4 Patient Subgroups, Based on Their Most Frequently Co-occurring Comorbidities, and Their Risk of Readmission (Red Text).

Drs Sharma and Raji inspected the network and noted that the readmission risk of the patient subgroups had a wide range (12.7%-19.6%) with clinical (face) validity. Furthermore, the co-occurrence of comorbidities in each patient subgroups was clinically meaningful. Drs Sharma and Raji inferred that patients in subgroup 1 had a low disease burden, with uncomplicated hypertension leading to the lowest risk for readmission (12.7%). This subgroup represented patients with early organ dysfunction who would benefit from creating checklists to remind clinicians to conduct procedures such as regular monitoring of blood pressure in predischarge protocols to reduce the risk of readmission. Next, Drs Sharma and Raji inferred that patients in subgroup 3 had mainly psychosocial comorbidities, which could lead to aspiration-precipitating pneumonia, in turn leading to an increased risk for readmission (15.9%). This subgroup would benefit from early consultation with specialists (eg, psychiatrists, therapists, neurologists, geriatricians) with expertise in psychosocial comorbidities, with a focus on the early identification of aspiration risks and precautions. Next, Drs Sharma and Raji inferred that patients in subgroup 2 had diabetes with complications, renal failure, and heart failure, and therefore had higher disease burden leading to an increased risk for readmission (17.8%) compared with subgroup 1. This subgroup had metabolic abnormalities with greater end-organ dysfunction and would therefore benefit from case management from advanced practice providers (eg, nurse practitioners) with rigorous adherence to established guidelines to reduce the risk of readmission. Finally, Drs Sharma and Raji inferred that patients in subgroup 4 had diseases with end-organ damage including gastrointestinal disorders, and therefore had the highest disease burden and risk for readmission (19.6%). This subgroup would also benefit from case management with rigorous adherence to established guidelines to reduce the risk of readmission. Furthermore, because patients in this subgroup typically experience complications that could impair their ability to make medical decisions, they should be provided with early consultation with a palliative care team to ensure that care interventions are aligned with patient preferences and values.

Congestive Heart Failure

The application of the inclusion and exclusion selection criteria described in Appendix A resulted in a training data set (n = 25 775 matched case-control pairs, of which 103 pairs of patients with no comorbidities were dropped) and a replication data set (n = 25 775 matched case-control pairs, of which 104 pairs of patients with no comorbidities were dropped), matched by age, sex, race, and Medicaid eligibility (a proxy for economic status). The feature selection described in Appendix B resulted in 42 unique comorbidities identified from the 3 comorbidity indices plus 1 condition-specific comorbidity. Of these, 1 comorbidity was removed because of <1% prevalence, and 37 comorbidities survived the significance and replication testing with Bonferroni correction. These data were used for the subsequent visual analytical modeling.

As shown in Figure 4, the bipartite network method identified 4 biclusters, each representing a subgroup of readmitted patients with CHF and their most frequently co-occurring comorbidities. The analysis revealed that the biclustering had significant modularity (Q = 0.17; z = 8.69; P = 3.636e−18) and significant co-occurrence replication (RI = 0.94; z = 17.66; P = 8.65e−70). Furthermore, as requested by the stakeholders, we superimposed the risk for each of the patient subgroups to enable a geriatrician to determine whether the subgroups were clinically meaningful.

Figure 4

Results From the CHF Network Analysis Showing 4 Patient Subgroups, Based on Their Most Frequently Co-occurring Comorbidities, and Their Risk of Readmission (Red Text).

Dr Raji inspected the network and noted that the readmission risk of the patient subgroups, ranging from 15.1% to 19.9%, was wide and had clinical (face) validity. Furthermore, the co-occurrence of comorbidities in each patient subgroup was clinically meaningful. He inferred that patients in subgroup 1 had chronic but stable conditions and therefore had the lowest risk for readmission (15.1%). Next, he inferred that patients in subgroup 3 had mainly psychosocial comorbidities but were not acute or fragile compared with subgroups 2 and 4 and therefore had medium risk (16.6%). Finally, he inferred that patients in subgroup 4 had severe chronic conditions; these patients were fragile and close to hospice care and were therefore at high risk for readmission (19.9%).

Total Hip/Knee Arthroplasty

The application of the inclusion and exclusion selection criteria described in Appendix A resulted in a training data set (n = 8249 matched case-control pairs, of which 1239 pairs of patients with no comorbidities were dropped) and a replication data set (n = 8249 matched case-control pairs, of which 1264 pairs of patients with no comorbidities were dropped), matched by age, sex, race, and Medicaid eligibility (a proxy for economic status). The feature selection described in Appendix B resulted in 39 unique comorbidities identified from the 3 comorbidity indices plus 2 condition-specific comorbidities. Of these, 11 comorbidities were removed because of <1% prevalence, and 11 that survived the significance and replication testing with Bonferroni correction. These data were used for the subsequent visual analytical modeling.

As shown in Figure 5, the bipartite network method identified 7 biclusters, each representing a subgroup of readmitted patients who had undergone THA/TKA and their most frequently co-occurring comorbidities. The analysis revealed that the biclustering had significant modularity (Q = 0.31; z = 2.52; P = .011) and significant co-occurrence replication (RI = 0.89; z = 3.15; P = .0016). Furthermore, as requested by the stakeholders, we superimposed the risk for each of the patient subgroups to enable a geriatrician to determine whether the subgroups were clinically meaningful.

Figure 5

Results From the THA/TKA Network Analysis Showing 7 Patient Subgroups, Based on Their Most Frequently Co-occurring Comorbidities, and Their Risk for Readmission (Red Text).

Dr Raji inspected the network and noted that patients who had undergone TKA were healthier, in general, than were patients who had undergone THA, and the network was therefore difficult to interpret when merged together. However, he noted that the range of readmission risk had clinical (face) validity. Finally, he noted that subgroups 2, 4, and 5 comprised patients with more-severe comorbidities related to lung, heart, and kidney, with higher risk for readmission, than patients in subgroups 1, 6, and 7, who had less severe comorbidities and therefore had a lower risk for readmission.

Chronic Obstructive Pulmonary Disease

Table 2 summarizes the percentage of patients correctly assigned to a subgroup by the classification model over 1000 iterations for the training (first row of the table) and testing components (second row of the table), including the total N, mean, SD, quantiles 0.025, 0.25, 0.50, 0.75, and 0.975, and the minimum and maximum. For the testing data, the percentage of patients correctly assigned to a subgroup ranged from 99.1% to 100.0%, with a median of 99.6%, and with 95% being in the range of 99.3% to 99.8%. A single model of all the data (not separated into training and testing components) correctly predicted subgroup membership for 99.9% of patients (14 443/14 457). Model coefficients are summarized in Appendix C.

Table 2

Percentage of Patients With COPD Correctly Assigned to a Subgroup by the Classification Model.

Congestive Heart Failure

Table 3 summarizes the percentage of patients correctly assigned to a subgroup by the classification model over 1000 iterations for the training (first row of the table) and testing components (second row of the table), including the total number; mean (SD); quantiles 0.025, 0.25, 0.50, 0.75, and 0.975; and the minimum and maximum. For the testing data, the percentage of patients correctly assigned to a subgroup ranged from 98.7% to 99.7%, with a median of 99.3%, and with 95% being in the range of 99.3% to 99.6%. A single model of all the data (not separated into training and testing components) correctly predicted subgroup membership for 99.2% of patients (25 476/25 672). Model coefficients are summarized in Appendix C.

Table 3

Percentage of Patients With CHF Correctly Assigned to a Subgroup by the Classification Model.

Total Hip and Total Knee Arthroplasty

Table 4 summarizes the percentage of patients who underwent a THA/TKA who were correctly assigned to a subgroup by the classification model over 1000 iterations for the training (first row of the table) and testing components (second row of the table), including the total number, mean (SD); quantiles 0.025, 0.25, 0.50, 0.75, and 0.975; and the minimum and maximum. For the testing data, the percentage of patients correctly assigned to a subgroup ranged from 99.4% to 100%, with a median of 99.9%, and with 95% being in the range of 99.8% to 100%. A single model of all the data (not separated into training and testing components) correctly predicted subgroup membership for 100% of patients (70107010). Model coefficients are summarized in Appendix C.

Table 4

Percentage of Patients Who Underwent THA/TKA Correctly Assigned to a Subgroup by the Classification Model.

Chronic Obstructive Pulmonary Disease

The application of the inclusion and exclusion selection criteria (described in Appendix A) resulted in a cohort of 186 041 patients (29 026 cases and 157 015 controls). As shown in Figure 6, the lumped CMS model had a C statistic of 0.624; this was not significantly different from the hierarchical model, which had a C statistic of 0.625. The 4 subgroup models had C statistics that ranged from 0.596 to 0.616. Appendix D provides additional details related to the distribution of risk among the subgroup models. Furthermore, as shown in Appendix D, the calibration plots revealed that all models had a slope close to 1 and an intercept close to 0. Finally, the hierarchical model had significantly higher NRI but not significantly higher IDI compared with the CMS reference model.

Figure 6

Results From the Prediction Modeling of the COPD Index Condition Showing the C Statistic of the Lumped CMS Model, the 4 Subgroup Models (SM-1, SM-2, SM-3, and SM-4), and the Hierarchical Model.

Congestive Heart Failure

The application of the inclusion and exclusion selection criteria (described in Appendix A) resulted in a cohort of 295 761 patients (51 573 cases and 186 935 controls). As shown in Figure 7, the lumped CMS model had a C statistic of 0.6; this was not significantly different from the hierarchical model, which also had a C statistic of 0.6. The 4 subgroup models had C statistics that ranged from 0.570 to 0.614. Appendix D provides additional details related to the distribution of risk among the subgroup models. Furthermore, as shown in Appendix D, the calibration plots revealed that all models had a slope close to 1 and an intercept close to 0. Finally, the hierarchical model had a significantly lower NRI and IDI than did the CMS reference model.

Figure 7

Results From the Prediction Modeling of the CHF Index Condition Showing the C Statistic of the Lumped CMS Model, the 4 Subgroup Models (SM-1, SM-2, SM-3, and SM-4), and the Hierarchical Model.

Total Hip and Total Knee Arthroplasty

The application of the inclusion and exclusion selection criteria (described in Appendix A) resulted in a cohort of 356 772 patients (16 520 cases and 321 441 controls). As shown in Figure 8, the lumped CMS model had a C statistic of 0.638; this was not significantly different from the hierarchical model, which also had a C statistic of 0.638. The 7 subgroup models had C statistics that ranged from 0.578 to 0.636. Appendix D provides the additional details related to the distribution of risk among the subgroup models. Furthermore, as shown in Appendix D, the calibration plots revealed that the lumped, CMS, and hierarchical models had a slope close to 1 and an intercept close to 0. However, the calibration plots also revealed that 4 subgroup models had poor calibration in terms of slope (0.868--1.176) and intercept (−0.427 to 0.396). Finally, the hierarchical model had a significantly higher NRI and IDI than did the CMS reference model.

Figure 8

Results from the Prediction Modeling of the THA/TKA Index Conditions Showing the C Statistic of the Lumped CMS Model, the 7 Subgroup Models (SM-1–SM-7), and the Hierarchical Model.

Figure 1. Observed Readmission Rates by Cluster (PDF, 124K)

Figure 2. Subgroup Model C-Statistics (PDF, 112K)

Figure 3. CMS Model C-statistics (PDF, 116K)

Figure 4. Box-plots of Predicted Probability of Readmission (PDF, 94K)

Figure 5. Calibration Plots Relating Observed Readmission Probability to the Logistic Quantile of Predicted Probability of Readmission (PDF, 108K)

Appendix D.1. Prediction Model Coefficients (PDF, 110K)

Figure 6. Observed Readmission Rates by Cluster (PDF, 113K)

Figure 7. Subgroup Model C-Statistics (PDF, 117K)

Figure 8. CMS Model C-statistics (PDF, 117K)

Figure 9. Box-plots of Predicted Probability of Readmission (PDF, 106K)

Figure 10. Calibration Plots Relating Observed Readmission Probability to the Logistic Quantile of Predicted Probability of Readmission (PDF, 122K)

Appendix D.2. Prediction Model Coefficients (PDF, 145K)

Figure 11. Observed Readmission Rates by Cluster (PDF, 118K)

Figure 12. Subgroup Model C-Statistics (PDF, 160K)

Figure 13. CMS Model C-statistics (PDF, 115K)

Figure 14. Box-plots of Predicted Probability of Readmission (PDF, 112K)

Figure 15. Calibration Plots Relating Observed Readmission Probability to the Logistic Quantile of Predicted Probability of Readmission (PDF, 113K)

Appendix D.3. Prediction Model Coefficients (PDF, 146K)