Background:

During routine clinical care, treatments are adaptive to patients' responses to previous treatment assignments. However, methods for comparative effectiveness research (CER) are predominately designed for nonadaptive treatments. This project aimed to evaluate the comparative effectiveness of patient-centered adaptive treatment strategies (PCATS) at the initiation of treatment and over the course of the disease progression. As a case in point, despite many medication options, polyarticular-course juvenile idiopathic arthritis (pcJIA) is often refractory and requires better adaptive treatment strategies (ATS).

Objectives:

Aim 1. To develop, refine, and disseminate bayesian causal inference methods for evaluating clinical effectiveness and for informing better PCATS.

Aim 2. To evaluate the clinical effectiveness of the recommended ATS for patients with pcJIA using real-world data.

Methods:

We propose the GPMatch method, a nonparametric full bayesian doubly robust causal inference method that uses Gaussian process (GP) prior as a matching tool. We performed simulation studies to evaluate its performance compared with that of some widely used causal inference methods: propensity score subclassification, augmented inverse treatment probability weighting, regression adjustment, and bayesian additive regression trees (BART), under dual-misspecification settings. We extended both GPMatch and BART methods for ATS and applied them to electronic medical record (EMR) data to compare 2 consensus treatment plans (CTPs) that began with a disease-modifying antirheumatic drug (DMARD) at different times in treating children with pcJIA: the early-combination plan uses biologic and nonbiologic DMARDs (b+nbDMARD) soon after diagnosis, while the step-up plan starts with an nbDMARD first and then introduces bDMARDs later. The primary end points were Clinical Juvenile Arthritis Disease Activity Score (cJADAS10, with a cutoff at 10 for active joint count) results at 6 and 12 months, and the secondary end point was the Pediatric Quality of Life Inventory (PedsQL) score at 12 months.

Results:

Simulation studies demonstrated that GPMatch, followed by BART, performed as well as or better than some commonly used non-bayesian causal inference methods for comparing both nonadaptive treatment strategies and ATS, as measured by the root mean square error (RMSE) and median absolute error (MAE). The pcJIA CER suggests that by 6 months, the early-combination plan reduced disease activity on average by 2.0 points (95% CI, 0.4-3.6 points) more than the step-up plan as measured by the cJADAS10. By 12 months, the early-combination plan remained more effective than the step-up plan: The average improvement in cJADAS10 was 2.6 points (95% CI, 0.6-4.6 points) if the first-line treatment was continued or reduced and 2.2 points (95% CI, 0.3-4.14 points) if the treatment was escalated. Both CTPs were effective in improving the PedsQL score by 12 months, reporting improvements of 74.8 ± 2.0 and 80.4 ± 3.7 points for the step-up and early-combination CTPs, respectively. If treated on the early-combination plan, patients were expected to achieve an average of 5.61 (95% CI, −3.89 to 15.12) more points on the PedsQL than were patients treated on the step-up plan.

Conclusions:

The GPMatch method accomplishes matching and flexible modeling in the same step and has well-calibrated frequentist properties. It is doubly robust in the sense that the average treatment effects are correctly estimated when either of the following conditions is satisfied: (1) The GP mean function correctly specifies the potential outcome model; and (2) the covariance function correctly specifies the matching structure. The pcJIA CER study suggests that the early-combination plan is more effective in reducing disease activity 1 year later. We developed a user-friendly graphic interface online R Shiny application, “PCATS,” which is easy to use, making GPMatch and BART methods accessible to general CER investigators.

Limitations:

The GPMatch method is computationally intensive and not yet extended to nonnormally distributed outcomes. The PCATS online app assumes no missing data and single time-dependent confounding. Missing data are an inherent feature of EMR data, and our CER study addressed missingness at the design, data management, and data analysis steps. Nevertheless, the study results may be limited by the missing data handling procedures. We assume that the EMR captures important treatment considerations from the clinician's perspective, but not from the patient's perspective. Sensitivity analyses were performed to account for missing potential confounders from the patient's perspective. Finally, the CER study only analyzed 2 of the 3 CTPs.

Motivation

Using causal inference methods, observational data can be used to provide evidence of clinical effectiveness.¹ However, most methods are designed to evaluate causal treatment effects from treatment assignments made at a single time point. Furthermore, the current statistical causal inference methodology for adaptive treatment strategies (ATS) may perform poorly under model misspecification.² However, when dealing with data collected from the real world, it is nearly impossible to correctly specify a model. We proposed a novel bayesian causal inference method that is designed to lessen the model misspecification problem in evaluating the comparative effectiveness of ATS.

In clinical practice, physicians adjust treatment plans over time and adapt them to patients' responses to previous treatments and disease progression. This is an example of an ATS. Not all time-varying treatments are ATS. A nonadaptive treatment strategy is a predetermined plan that does not adapt to patients' responses to previous assignments. The use of ATS is ubiquitous in clinical care practices,³ yet currently, clinical decisions are predominantly based on evidence provided by parallel-arm randomized clinical trials, whose results do not necessarily apply to the adaptive treatment. Although a sequential multistage adaptive randomized trial (SMART) could be used, it may not be feasible for rare disease- or low disease-prevalence settings.

This project was motivated by the need to evaluate the comparative effectiveness of ATS that are common in treating patients with chronic disease. One of the specific aims of the project was to evaluate a set of time-varying adaptive consensus treatment plans (CTPs) recommended for children with juvenile idiopathic arthritis (JIA).⁴

JIA is one of the most common types of rheumatologic disease in children. The cause of childhood arthritis is unknown, and current understanding of the disease etiology and pathogenesis is limited.⁵ The prevalence rate of JIA is approximately 19.4 per 100 000 for girls and 11.0 per 100 000 for boys.⁶ JIA is a heterogeneous group of diseases. Systemic JIA presents distinct features and requires more distinctive treatment approaches than do other types of JIA. Nonsystemic JIA includes polyarthritis, oligoarthritis, psoriatic arthritis, enthesitis-related arthritis, and undifferentiated arthritis, collectively referred to as polyarticular-course JIA (pcJIA). pcJIA is often refractory to treatment, and patients with pcJIA alternate between relapse and remission.⁷

Various treatment options have been made available for JIA in the past 2 decades.⁸ The advent of biologic disease-modifying antirheumatic drugs (bDMARDs) and nonbiologic disease-modifying antirheumatic drugs (nbDMARDs) has revolutionized treatment, making it possible to set inactive disease as the treatment target.⁹ However, it is unknown at the time of initial treatment which treatment strategy will be the most effective to induce remission for a given individual. Additionally, for a patient who does not respond to previous treatment, the next best option is often unknown. Such poorly guided treatment strategies produce inferior treatment outcomes.⁴ Despite advanced medical treatment, half of patients experience suboptimal health-related quality of life (HRQOL).¹⁰ As the first step towards informing better medical decision-making to help optimize patient outcomes, a panel of JIA experts developed 3 CTPs (Figure 1): step up, early combination, and biologic only for patients newly diagnosed with pcJIA.⁴

Figure 1

CTPs for Patients With pcJIA.

In this project, we intended to answer the following specific comparative effectiveness research (CER) questions:

1. At the time of pcJIA diagnosis, is early combination of biologic and nonbiologic DMARD (b+nbDMARD) treatment more effective in improving the Clinical Juvenile Arthritis Disease Activity Score (cJADAS10, with a cutoff at 10 for active joint count) at 6 months than the commonly used nbDMARD treatment?
2. After completion of the first stage of treatment at 6 months, given patients' responses to the previous treatment assignment, is it more effective to adapt or not to adapt treatment following the CTPs?
3. Over 12 months of treatment following the CTPs, what are the differences in the effectiveness of ATS?

Need for Methods Development

Causal inference using observational data rests critically on the concept of the potential outcomes. To understand the potential outcome, we may imagine a counterfactual world, where the same patient could be treated multiple times using different treatments started at the same time when a treatment decision needs to be made. The outcome that we would observe if we could have access to this counterfactual world, under all potential treatment choices, is the potential outcome. For example, if there are 2 potential treatment choices, then there are 2 potential outcomes, Y, which we may denote by Y⁽⁰⁾ and Y⁽¹⁾. Then, we could easily identify the best treatment choice for the patient by comparing Y⁽¹⁾ against Y⁽⁰⁾. However, the counterfactual world is not real. In the real world, we could only observe 1 of the potential outcomes, that is, Y = Y⁽⁰⁾ if treated with the comparator, and Y = Y⁽¹⁾ if treated with the intervention under investigation. The fundamental challenge in causal inference is to uncover the missing potential outcomes.

For ATS, we are interested in comparing treatment sequences. Specifically, for comparing the early-combination vs step-up CTPs presented in Figure 1, we are comparing 4 possible treatment sequences: (1) b+nbDMARD followed by nbDMARD, (2) nbDMARD followed by b+nbDMARD, (3) b+nbDMARD throughout, and (4) nbDMARD throughout. At the 6-month time point following the first stage of treatment, we have 2 potential outcomes: Y⁽⁰⁾ from treatment with nbDMARD at baseline, and Y⁽¹⁾ from treatment with b+nbDMARD at baseline. At the 12-month time point following the decision made at 6 months about the second stage of treatment, we have 4 potential outcomes, Y⁽⁰⁰⁾ and Y⁽¹¹⁾ from treatment with nbDMARD and the b+nbDMARD throughout both stages, respectively; and Y⁽¹⁰⁾ and Y⁽⁰¹⁾ from treatment with bDMARD early in stage 1 and later in stage 2, respectively. As this example shows, the fundamental challenge of causal inference is even more difficult in the ATS setting.

The complexity of ATS requires advanced causal inference methods. Because of the adaptive assignment process, patients who respond better (or worse) are likely to end up in the same arm of ATS; thus, the treatment effects are confounded by the posttreatment time-varying intermediate outcomes and covariates, in addition to the baseline covariates. The propensity of treatment assignment differs at every decision point, based on time-varying covariates, treatment history, and disease progression. Any misspecification in these propensity scores could be propagated. In addition, the number of potential outcomes exponentially increases with the increased stages of ATS, leading to increasingly sparse data and underpowered causal inference. Therefore, model misspecification is particularly challenging under the ATS setting. To ensure that CER produces valid, reliable, and reproducible results for the ATS, it is critical that causal inference methods be robust to model misspecification and able to address time-varying confounding.

The most widely used causal inference method is the propensity score (PS) method. A Google Scholar search of the keyword “propensity score” alone revealed >170 000 scholarly works since 2010. Despite the theoretical appeal of PS approaches, the validity of CER results hinges on the correct specification of the PS.¹¹ Many other causal methods have been proposed, which fall into 3 categories: design based (eg, PS and matching methods), model based (eg, bayesian nonparametric), and a combination of design- and model-based methods. Most of these methods rely on strong causal inference assumptions, such as no unmeasured confounders and correct model specification. Comprehensive evaluations of the existing methods suggest poor operating characteristics of some commonly used causal inference methods. For example, increasing sample size could lead to more biased results under the most realistic setting of dual misclassification (ie, we do not know the true model behind the data-generating processes of either the exposure or the outcome).^11,12 As a result, CER studies of ATS could lead to inconsistent results and misleading conclusions.

Doubly robust (DR) approaches attempt to address model misspecification. When the PS model is misspecified, a DR estimator produces valid causal inference if the outcome modeling is correctly specified. Most DR methods use some combination of PS and outcome modeling. The most widely adopted method is the augmented inverse probability treatment weighting (AIPTW), which augments the inverse probability treatment weighting (IPTW) by separate outcome modeling. Incorporating the PS or a function of the PS into the outcome regression modeling as a covariate is another approach.¹³ Comprehensive studies by Gutman and Rubin¹² suggest that the performance of widely adopted methods, such as PS matching, may suffer poor operating characteristics (ie, the average treatment effect [ATE] estimates do not approach the truth as sample size increases). Considering a single confounding variable X, the authors suggest that regression of outcome Y ∼ X after matching on X provides a better solution. However, there are many different matching approaches that can vary in performance. Many matching procedures require arbitrary decisions on the caliper, a parameter to determine whether a match is achieved, as well as whether matching is done with or without replacement and the matching ratio. Sometimes, the matching procedures lead to discarding a large percentage of individuals, which can limit the generalizability of the results and reduce study power.

The performance of existing methods for ATS has not been adequately evaluated under realistic problem setups. In the current literature, studies evaluating the performance of causal inference methods have been predominately performed under nonadaptive treatment settings. Only a few studies have evaluated ATS. Daniel et al¹⁴ provide a tutorial and comparison of different ATS methods under the randomized trial setting (ie, the SMART setting). Newsome et al² presented a comparative study of existing ATS methods in an observational study setting. These studies did not consider potential model misspecification for outcome models. Furthermore, we lack general and easy-to-use analytic tools for ATS studies that do not demand specialized statistical programming skills and that are accessible to the general research community.

Causal Assumptions

Most existing causal inference methods rely on 3 fundamental causal assumptions¹³:

• Stable unit treatment value assumption (SUTVA). The potential outcomes of 1 experimental unit do not change despite how the treatment was assigned and are not related to the treatment received for the other experimental units.
• Strong ignorable treatment assignment assumption. The treatment assignment (A) is independent from the potential outcomes (Y⁽⁰⁾, Y⁽¹⁾) given the measured confounders. In other words, there are no unmeasured confounders in the study.
• Positivity assumption. This assumption ensures that every patient has a nonzero probability of being assigned to one of the treatment arms.

These causal assumptions are required within the PS framework, but they were widely considered overly strong. It is conceivable that outcome measures may be subject to measurement error. In addition, many factors may influence the treatment effect, such as presurgery procedures, timing of treatments, and concomitant medications. These factors are likely to subject the observed outcomes to additional measurement error. In addition, the unmeasured confounder assumption is widely acknowledged as overly strong for observational studies. These widely adopted assumptions were laid out within the theory of the PS framework. Alternative frameworks for causal inference were proposed under the direct acyclic graphic (DAG) framework¹⁵ and more recently under a stratified-sampling framework.¹⁶ In the Methods section, we present an alternative and weaker (ie, more easily met) set of causal assumptions.

Advantages and Challenges of Bayesian Causal Inference

Bayesian modeling is particularly suitable for CER because it can easily incorporate existing knowledge into a prior distribution, synthesize data evidence from different sources, account for model uncertainties, and inform optimal decisions.¹⁷ It produces a posterior distribution that offers more information beyond the traditional point estimate and 95% CIs. The parameter-rich bayesian modeling techniques provide a powerful tool for addressing potential model misspecifications. A full bayesian approach offers a coherent and versatile framework to address time-varying adaptive treatment assignment and time-varying confounding.

Bayesian approaches to causal inference have primarily taken the direct modeling approach. The direct modeling of outcomes allows us to use the many well-developed regression modeling techniques, including parametric to nonparametric approaches, and to address complex data types and structures. The bayesian approach allows for incorporating prior knowledge and synthesizing information from different sources and thus can be used for tackling complex problems involving encouragement trials,¹⁸ dynamic treatment regimes,¹⁹ and treatment noncompliance.^20,21 More recently, bayesian nonparametric approaches have been increasingly used. Bayesian additive regression trees (BART) have been shown to produce more accurate estimates of ATEs than do PS matching, propensity-weighted estimators, and regression adjustment in the nonlinear setting, and they perform as well in the linear setting.^22,23 Gustafson suggested the use of the weighted average of the answers from a parametric and a nonparametric bayesian model.²⁴ Others have advocated for the use of bayesian model averaging, including Cefalu et al²⁵ and Zigler and Dominici.²⁶ More recently, bayesian modeling with Gaussian process (GP) and Dirichlet process priors has been used to address heterogeneity of treatment effects (HTE),²⁷ dynamic treatment assignment,²⁸ and missing data.²⁹ These parameter-rich models could mitigate concerns over potential model misspecification. However, by not accounting for confounding-by-indication in the analyses, the parametric-rich model could suffer from overfitting, which subsequently may introduce bias in estimating causal treatment effects.

A better bayesian causal inference method is needed to account appropriately for bias due to confounding-by-indication in observational studies. Currently, the bayesian approach in causal inference is predominately model based, imputing the missing potential outcomes. However, causal inference presents challenges in addition to the conventional missing data problem.³⁰ In particular, because no more than 1 potential outcome can be observed for a given individual, the missing data are highly structured such that the correlations between the 2 potential outcomes are nonidentifiable. Consequently, different analyses could arrive at different inferential results; this is the issue of “inferential quandary.”³¹ Confounding-by-indication and time-dependent confounding are additional challenges.³⁰ Ignoring these challenges may lead to biased estimates of causal effect.³² Many investigators have been actively searching for ways to correct for treatment selection bias in bayesian causal inference. Including the PS as a covariate in the outcome regression model is 1 option. However, joint modeling of outcome and treatment selection models leads to a “feedback” issue, in which the information from the outcomes plays an important role in the estimation of PS such that it defeats the role of PS as a balancing score, and this subsequently leads to biased causal inference estimates. To overcome the feedback issue, a 2-stage approach was suggested.^33,34 Saarela et al³⁵ proposed an approximate bayesian approach incorporating inverse treatment assignment probabilities as importance sampling weights in Monte Carlo integration. This approach offers a bayesian version to AIPTW. More recently, Hahn et al³² suggested introducing the PS into the formulation of a prior as a way of regularizing outcome modeling. These methods all require a 2-stage approach, and their performance may suffer if the PS is not correctly specified.

Objectives

Motivated by the need to fill the gap in the understanding of the effectiveness of CTPs in patients with pcJIA and recognizing the need for methodology development in bayesian causal inference, we proposed the patient-centered adaptive treatment strategies (PCATS) project. This project had 2 specific aims:

Aim 1. To develop, refine, and disseminate bayesian causal inference methods for evaluating clinical effectiveness and for informing better PCATS.

Aim 2. To evaluate the clinical effectiveness of the newly recommended ATS for patients with pcJIA via an analysis of real-world data.

Significance of the Project

Routine clinical approaches in treating patients with chronic or prolonged disease conditions are time dependent and adaptive to patient responses to the previous treatment, yet few causal inference methods have been developed for ATS. A robust and efficient causal inference method that can be used for evaluating the comparative effectiveness of ATS is essential to inform better treatment strategies for patients with chronic or prolonged disease conditions. No user-friendly statistical software tool is available for evaluating causal treatment effect for ATS. This project is important because it filled this methods gap. We developed a novel bayesian DR causal inference method, GPMatch. We also offer a user-friendly online application with a graphic user interface (GUI), “PCATS,” which makes advanced bayesian causal inference methods (both GPMatch and BART) for both nonadaptive treatment strategies and ATS available and accessible to the general public. In addition, the project applied these advanced causal inference methods to evaluate the CER of 2 ATS in treating children with newly diagnosed pcJIA. The CER study is significant, as it offers, for the first time, real-world evidence of the effectiveness of the early-combination plan vs the step-up treatment plan.

Partnering With PR-COIN

We actively engaged patients and stakeholders through PR-COIN (https://pr-coin.org/), a member of PCORnet. PR-COIN, which currently has 18 participating clinical centers across North America, holds semiannual learning sessions for patients/parents and health care providers to share ideas and learning experiences for improving outcomes in patients with JIA. We shared the PCATS study results and received input from patients and health care providers during these learning sessions. We also discussed topics such as study design, rigorous statistical analyses, and data quality issues. Additionally, this partnership has helped PR-COIN with improving data quality and data collection forms.

Stakeholder Advisory Panel

The SAP consisted of 2 parents of patients, 2 clinicians, and 1 health policy researcher. The primary mission of the SAP was to ensure that the project was patient centered—in particular, to confirm that the methods development could adequately answer the specific patient-centered questions and that the methods development was understandable, meaningful, and generalizable to broader patient populations and stakeholders. The secondary mission of the SAP was to monitor and advise on project progress, provide approval, and facilitate dissemination of the project results. The principal investigator, project manager, and 5 SAP members had 1 in-person meeting or 2 conference calls every year.

GP Covariance as a Matching Function

Matching experimental units on their pretreatment assignment characteristics helps remove bias by ensuring similarity or balance between the experimental units in the 2 treatment groups. Matching methods impute the missing potential outcome with the value from the nearest match or the weighted average of the values within the nearby neighborhood defined by (a chosen value) caliper. Matching on multiple covariates can be challenging when the dimension of the covariates is large. For this reason, matching is often performed using the estimated PS or by the Mahalanobis distance (MD). Under the no-unmeasured confounder setting, matching induces balance in covariates between the treated and untreated groups. Therefore, it serves to transform a nonrandomized study into a pseudorandomized study. There are many different matching techniques.³⁷ A recent simulation study by King and Nielsen³⁸ compared PS matching with MD matching. Their study showed that, under some settings, PS matching can result in more biased and less accurate estimates of averaged causal treatment as the precision of matching improves, while the MD matching showed improved accuracy.³⁸ A common practice in matching, particularly in 1:m matching, is that individuals without a match are discarded. Such a practice may lead to a sample that is no longer representative of the target population and might reduce study power. A user-specified caliper is often required, but it is not immediately clear how to best choose an optimal caliper. Furthermore, matching on a misspecified PS may lead to biased causal inference results. A better approach to matching should be nonparametric and avoid arbitrary decisions on caliper or discarding data points.

GP prior has been widely used to describe biological, social, financial, and physical phenomena, due to its ability to model highly complex dynamic systems and its many desirable mathematical properties. Recent literature (eg, Choi and Woo³⁹ and Choi and Schervish⁴⁰) has established posterior consistency for bayesian partially linear GP regression models. Bayesian modeling with a GP prior can be viewed as a marginal structural model where the potential outcome under the no-treatment condition is modeled nonparametrically. It allows for predicting the missing potential outcomes of a given patient by a weighted sum of the observed data from his or her matched neighbors, with larger weights assigned to those neighbors in closer proximity and smaller weights to those neighbors further away from the patient, much like a matching procedure.

To improve causal inference methods that are robust to model misspecification and allow the bayesian causal inference to account for confounding-by-indication within a full bayesian framework, GPMatch is proposed as a bayesian nonparametric causal inference method. GPMatch uses GP prior as the matching tool, where the GP prior is formulated in such a way that for each individual patient (i-th) in the sample, it allocates a weight range from 0 to 1 to the outcome observed from other (j-th) patients in the data set, based on their similarity defined by the squared-exponential covariance matrix:

Here, v_ki and v_kj are observed values of the k-th confounding variable for the i-th and j-th patients, correspondingly. The length-scale parameters ϕ_k and variance ${\sigma }_f^2$ determine the smoothness and shape of the biologic mechanism, which are estimated based on the observed data.

The SE covariance function is used for its ability to fit the smoothed response surface. By including confounding variables (denoted by V) into the covariance function, the GP prior specifies that patients with the same values of all confounding variables are matched completely, that is, assigned a weight of 1. The matching utility of GP prior can be considered as a matching process being performed for each i-th patient. The K(v_i, v_j) determines how similar or dissimilar the j-th patient is compared with the i-th patient. It assigns a larger weight to patients who are similar, and it assigns less or 0 weight to patients who are less similar or sufficiently different. As a consequence, GP prior accomplishes “matching” for each individual patient. Of note, although only a small data set is used for matching the i-th patient, different sets are used for different patients. Collectively, no data are discarded, and all data are used for estimating the causal treatment effect.

After matching, GPMatch then estimates the expected potential outcomes for a given patient by using data from other matched patients who are sufficiently similar. The matching, weighting, and estimation processes are accomplished in a single step of bayesian GP regression modeling. The GPMatch method can easily incorporate different types of treatments. For example, continuous treatment and its potential higher-order terms could be included in modeling treatment. Heterogeneous treatments can be evaluated by including treatment-by-covariate interactions. Higher-order terms could be included to model a treatment effect as a nonlinear function of a continuous variable.

The SE distance can be considered as an alternative metric to the MD,

where c ∈ R⁺ is the caliper and S is the sample variance-covariance matrix of confounding variables v. Of note, MD matching requires specification of a caliper. Smaller c leads to more precise matching but often results in a serious reduction in sample size after matching. Compared with MD matching, GPMatch does not require arbitrary specification of a caliper; instead, the length-scale parameters (ϕ_k), which govern the extent to which the data points are matched, are estimated from the data. We allow different length-scale parameters for different confounding variables, such that they acknowledge that some confounders may play a relatively more important role in matching than other confounders. The variables with larger values of ϕ_k parameters are considered more important than those with smaller values. GPMatch produces valid causal inference results under some important causal assumptions.

Causal Assumptions

Nonadaptive Treatment

Considering 1-time-point treatment assignment, the causal assumptions are depicted in the DAG (Figure 3), where rectangular nodes are observed variables and oval nodes are latent or unobserved variables. For simplicity of presentation, we describe the assumptions by considering a binary treatment assignment A = 0/1, where 0 indicates control and 1 indicates intervention. The method, however, is applicable to any type of treatment. Corresponding to A = (0, 1), the potential outcomes (Y⁽⁰⁾, Y⁽¹⁾) are 2 latent variables. The unobserved covariates are denoted by U. The potential Y⁽⁰⁾ under the controlled condition is determined jointly by X, a p-dimensional vector, and V, a q-dimensional vector, of the observed covariates plus an unobserved covariate, U₀. Thus, (X, V, U₀) are prognostic variables. Similarly, the potential outcome Y⁽¹⁾ under the intervention condition is determined jointly by the observed covariates (X, V) and the unobserved covariates (U₀, U₁). The observed outcome Y is a noisy version of the corresponding potential outcomes, with error term (ε). The treatment is assigned according to an unknown PS, which is determined by the baseline covariates: observed V and unobserved U₂. The observed baseline covariates (X, V) could be overlapping sets, whereas different symbols are used to distinguish their roles in biologic mechanisms driving potential outcomes and the treatment assignment process. For example, X could include patient age, sex, genetic makeup, family disease history, and past and current medication use, as well as laboratory results and other disease characteristics, which are directly related to the prognosis of the disease. The V could include the above-mentioned X variable, as well as other considerations of the treatment decision, including patient insurance, socioeconomic status, education, and clinical centers. Most of these important X and V covariates are available in disease registries or electronic medical records (EMRs). Other factors, such as patient and clinician personal preferences, cultural beliefs, and past experiences, may play a role in treatment decisions. However, these factors are almost never recorded, and are collectively referred to as U₂. The DAG could also include additional paths among (U₀, U₁, X, V) to allow for correlations among them. These paths are not included in Figure 3 to provide a clearer visual presentation. The direction of the arrow in the DAG indicates the direction from a cause or determinant to an effect. For example, the observed Y is directly determined by the treatment assignment (A) and the potential outcomes Y⁽⁰⁾ and Y⁽¹⁾. For another example, the V → A path suggests that V encompasses all the observed direct determinants to the treatment decision. The bidirectional arrow indicates the correlated relationship.

Figure 3

DAG Presentation of the Counterfactual World-Problem Setup.

A side-by-side comparison of the DAG with the 3 widely adopted causal assumptions laid out by Rosenbaum and Rubin¹³ should be helpful to see the differences between the two.

Causal Assumption 1

Instead of the SUTVA, we make the stable unit treatment value expectation assumption (SUTVEA). Specifically:

• The consistency assumption by Rosenbaum and Rubin¹³ requires that the observed outcome is an exact copy of the potential outcome, that is, $Y_i=Y_i^{(0)}(1-A_i)+Y_i^{(1)}{A}_i$. Instead, we consider the observed outcome as a noisy version of the potential outcome, where expectation of the observed outcome $E(Y_i)=Y_i^{(0)}(1-A_i)+Y_i^{(1)}{A}_i$.
• The no-interference assumption by Rosenbaum and Rubin¹³ requires that the potential outcomes of 1 unit are not affected by the treatment of other units. Instead, we assume that the observed outcomes from different units are conditionally independent given the observed covariates $Y_i^{(a)}\perp Y_j^{(b)}|X,V$.

The SUTVEA assumption acknowledges the existence of residual random error in the outcome measure. The observed outcomes may differ from the corresponding true potential outcomes due to some measurement error. In addition, the observed outcomes could differ when the treatment received deviates from the intended version of treatment. For example, outcomes could differ by the timing of the treatment, the presurgery preparation procedure, or the concomitant medication.

A hypothetical example may help illustrate the subtle difference in the 2 no-interference assumptions. Jane (i) and Joe (j) are a wife and husband who live together. Jane always cooks. Jane takes a certain medication (A_i = 1) for hormonal changes (U₂), but Joe does not (A_j = 0). The hormonal changes in Jane lead her to crave fatty food (V). The fatty diet may causally increase cholesterol (Y) for both Jane and Joe. This is an example of the no-interference assumption being violated because the cholesterol level in Joe is related to medication use by Jane. Here, the cholesterol levels in Jane and Joe are correlated, due to the unobserved U₂. However, U₂ had an effect on the potential outcomes only via X (eg, sex, cohabitation, and race) and V. Here, ($Y_i^{(a)}$, $Y_j^{(b)}$) are correlated, while $Y_i^{(a)}\perp Y_j^{(b)}|X,V$. In other words, the correlation between Jane's and Joe's potential outcomes is determined by their diet and other covariates X. Conditional on (X, V), the treatment received by Jane and, subsequently, her potential outcome, is independent from the potential outcome for Joe.

Causal Assumption 2

Approaching causal inference as a missing potential outcome problem, we require a missing-at-random (MAR) assumption for the joint distribution of the outcomes; that is,

$$\begin{array}{l}[Y^{(0)},Y^{(1)}|A=1,X,V]=[Y^{(0)},Y^{(1)}|A=0,X,V],\text{\hspace{0.17em}and}\\ [Y|Y^{(0)},Y^{(1)},A=1,X,V]=[Y|Y^{(0)},Y^{(1)},A=0,X,V].\end{array}$$

Jointly, we may write

$$[Y,Y^{(0)},Y^{(1)}|A=1,X,V]=[Y,Y^{(0)},Y^{(1)}|A=0,X,V].$$

Here the [.] notation indicates the joint distribution. This is equivalent to the MAR assumption that is widely adopted in the missing data context. The assumption is necessary to ensure that the causal effect is identifiable. It does not require the unmeasured-confounder assumption. Rather, it only requires that the minimum sufficient set be observed following the DAG. It allows for 3 types of confounders (U₀, U₁, U₂). Although (U₀, U₁, U₂) are correlated with both (Y) and (A), we can see from the DAG that their existence does not affect the identifiability of the causal treatment effect and thus is admissible. If U₂ is null, then the assumption is equivalent to the strong ignorable treatment assignment assumption.

Causal Assumption 3

Positivity Assumption

As with Rosenbaum and Rubin,¹³ we assume every sample unit has a nonzero probability of being assigned to either of the treatment arms, that is, 0 < Pr(A_iǀV_i) < 1.

This assumption is adopted to ensure the equipoise of the causal inference. Because we can never tell whether the lack of overlap in covariates is a manifestation of data sparsity or a lack of equipoise, we assume that positivity is ensured at the design stage rather than at the analytical stage.

Estimating Average Causal Treatment Effect

Considering the 1-stage setting, GPMatch fit the outcome Y by a marginal structural model:

for i = 1, …, n, where f(.) ∼ GP(0,.), and ${\epsilon }_{i}iid\sim N(0,{\sigma }_0^2)$. Without loss of generality, we assume that covariates X are a subset of V. We may let $\tau (x)={\mathit{x}}_{\mathit{i}}^{\mathit{T}}\mathit{\beta }$, which allows for an estimation of the conditional average causal treatment effect (CATE) given X. Letting Y = (Y₁, …, Y_n), we may rewrite equation 1 by a multivariate representation of

where $\mathit{m}={(A_i\times {\mathit{X}}_{\mathit{i}}^{\mathit{T}}\mathit{\beta })}_{\mathit{n}\times \mathbf{1}}$, $\mathbf{\Sigma }={\left({\sigma }_{ij}\right)}_{n\times n}$, ${\sigma }_{ij}=K(v_i,v_j)+{\sigma }_0{\delta }_{ij}$,

for i, j = 1, …, n. The (ϕ₁, ϕ₂, …, ϕ_q) are the length-scale parameters for each of the covariates v. The δ_ij is the Kronecker function, δ_ij = 1 if i = j, and 0 otherwise. The covariance function K(v, v′) = Cov(v, v′) models the covariance structure between any 2 inputs (v, v′). The 2 data points are increasingly correlated as they move closer in proximity within the covariate space.

To answer the CER questions that motivated the study, we focused on estimating the following average causal treatment effect.

1. The ATE at stage 1 (ATE@stage1): $\widehat{E}(Y_1^{(1)}-Y_1^{(0)}|X_0)$
2. The ATE at stage 2 conditional on the past treatment assignment and the patient's responses (CATE@stage2): $\widehat{E}(Y_2^{(a_0,1)}-Y_2^{(a_0,0)}|X_0,X_1(a_0)=x_1,Y_1(a_0)=y_1)$
3. The marginal ATE (MATE) at the end point is marginalized over the intermediate responses (MATE@stage2): $E\text{(}{Y}^{(a_0,a_1)}-Y^{(a_0^{\prime },a_0^{\prime })}|X_0)$

Box 1 presents the GPMatch algorithm for estimating the average causal treatment effect and the potential outcomes $Y_i^{(a)}$. Further technical details can be found in our methodology manuscript (Appendix B).

For evaluating the causal treatment effects of time-varying adaptive treatments, the GPMatch approach can be easily extended following the bayesian g-computation formula approach. Similarly, the BART⁴¹ method could be extended for the ATS. Under the causal assumptions described above, the g-computation formula factorizes the joint likelihood of all outcomes into a product of multiple conditional likelihoods of outcome models at each of the follow-up time points, given the history of treatment and covariates, up to the final study end point, for k = 1,2,…,K. Similarly, the g-computation formula can be used in conjunction with BART for the ATS.⁴² The estimates of the bayesian nonparametric model are used to predict the missing potential outcomes at each decision point in a sequential generative model. Thus, the potential outcomes for any given treatment history are predicted, and the ATE is estimated by the contrast between an intervention and the comparator ATS at the final study end point. Finally, the optimal ATS can be identified by maximizing the potential outcomes. Box 2 outlines the algorithm used for a 2-time-point treatment assignment.

Simulation Studies for 1-Time-Point Treatment Assignment

Under the dual-misspecification setting, 3 sets of simulation studies with 1-time-point binary treatment assignment design were considered to (1) evaluate its frequentist performance, (2) compare with the MD matching method, and (3) compare with the widely adopted methods. Below, we present a brief summary of the results and simulation settings. The detailed results are reported in Appendix B.

Setting 1: Well-Calibrated Frequentist Performance

To calibrate with frequentist performance, we considered a single covariate, x ∼ N(0, 1). The potential outcome was generated by y^(a) = e^x + (1 + U) × a + U₀ for a = 0, 1, where the true treatment effect was 1 + U_i for the i-th individual unit. The (U, U₀) are unobserved covariates. The treatment was selected for each individual following logit (P(A = 1|X)) = −0.2 + (1.8X)^1/3. The observed outcome was generated by $\text{y}|\text{x},\text{\hspace{0.17em}a}\sim \text{N}({\text{y}}^{(\text{a})},{\text{σ}}_0^2)$. We considered 4 parameter setups that involved 4 different random errors in (1) potential outcome Y(0), ${\text{U}}_0\sim \text{N}(0,{\text{\hspace{0.17em}γ}}_0^2)$; (2) treatment effect Y⁽¹⁾ − Y⁽⁰⁾, ${\text{U}}_1\sim \text{N}(0,{\text{\hspace{0.17em}γ}}_1^2)$; (3) treatment probability, ${\text{U}}_2\sim \text{N}(0,{\text{\hspace{0.17em}γ}}_0^2)$; and (4) observed outcome Y, ${\text{U}}_3\sim \text{N}(0,{\text{\hspace{0.17em}γ}}_3^2)$. Setting 1 included random error in both potential outcomes Y⁽⁰⁾ and the observed outcome, and setting 2 included random error in potential outcomes Y⁽⁰⁾ and the treatment effect. Settings 3 and 4 add to settings 1 and 2 another random effect in the treatment propensity. The simulation study shows that the approach performed well with respect to frequentist properties (Figure 4). It performs better than the AIPTW, propensity score subclassification by quintiles, and g-estimation methods, and it performs as well as linear regression modeling with spline fit propensity score adjustment and BART (Figure 5). The additional results, including a comparison of the GPMatch results to the gold standard (Table 1 in Appendix B) and other widely used methods (Tables S1-S4, Figures S1-S4), are presented in Appendix B.

Figure 4

Distribution of the GPMatch Estimate of ATE by Different Sample Sizes Under the Single Covariate Simulation Study Setting.

Figure 5

Comparisons of RMSE and MAE of the ATE Estimates by Different Methods Across Different Sample Sizes.

Setting 2: GPMatch Compared With MD Matching

To compare the performances of MD matching and GPMatch, we considered a simulation study with 2 independent covariates x₁, x₂, from the uniform distribution U (−2, 2), where treatment was assigned by letting A_I ∼ Ber(π_i), where

$$logit{\pi }_i=-x_1-x_2.$$

The potential outcomes were generated by

$$\begin{array}{c}{y}_i^{(a)}=3+5a+x_{1i}^3,\\ Y_i|X_i,A_i\sim N(y_i^{(Ai)},1).\end{array}$$

The true treatment effect was 5. Three different sample sizes were considered, N = 100, 200, and 400. For each setting, 100 replicates were performed, and the results were summarized.

Figure 6 presents the bias of the ATE estimate results comparing GPMatch (the horizontal short dashed line is the averaged ATE, and the 5th and 95th percentiles are the long dashed lines) with the MD match (circles are the point estimate, and vertical lines are the 95% CI estimates corresponding to different choices of caliper). The results clearly demonstrate better accuracy and efficiency using GPMatch.

Figure 6

Simulation Study Results From Comparison of GPMatch With MD Matching.

Setting 3: Performance Under Dual Misspecification

Following the well-known Kang and Schafer dual-misspecification simulation setting,¹¹ covariates z₁, z₂, z₃, and z₄ were independently generated from the standard normal distribution N(0, 1). Treatment was assigned by A_i ∼ Ber(π_i), where

$$logit{\pi }_i=-z_{i1}+0.25z_{i3}-0.1z_{i4.}$$

The potential outcomes were generated for a = 0, 1 by

$$\begin{array}{c}{y}_i^{(a)}=210+5a+27.4z_{i1}+13.7z_{i2}+13.7z_{i3}+13.7z_{i4},\\ Y_i|A_i,X_i\sim N(y^{(A_i)},1).\end{array}$$

The true treatment effect was 5. To assess the performance of the methods under dual misspecification, the transformed $x_1=\exp \left(\frac{z_1}{2}\right)$, $x_2=\frac{z_2}{1+\exp (z_1)}+10$, $x_3={(\frac{z_1{z}_3}{25}+0.6)}^3$, and x₄ = (z₂ + z₄ + 20)² were used in the model instead of z_i.

We compared GPMatch with many widely adopted causal inference methods. Here, we considered 2 different modeling strategies. In GPMatch1, the mean function included only the treatment effect. In GPMatch2, the mean function also included the X₁ − X₄. The results (Figure 7) suggest that GPMatch clearly outperformed other methods.

Figure 7

RMSE and MAE of ATE Estimates Using Different Methods Under the Kang and Schafer Simulation Setting.

Simulation Studies for ATS

Because the primary goal of the study was to evaluate the performance of 2-stage BART and GPMatch for estimating the ATE for ATS, 4 simulation designs were considered a SMART, where the treatment is randomly assigned at the first stage. The second-stage treatment is assigned adaptively to the patient's responses to the previous treatment. The design also considered the HTE setting. Five sets of simulation studies are specified below.

Setting 1: SMART Nonlinear Model

This simulation resembles a SMART setting, where the initial treatment is randomly assigned (Figure 8). The stage 1 treatment has no effect on the disease progression at the end of first-stage treatment. Both models of treatment assignment at the second stage and the potential outcomes are nonlinear functions of the end of the first-stage responses L₁,

$$\begin{array}{c}X\sim Bernoulli(0.4),A_0\sim Bernoulli(0.5),L_1(a_0)\sim N(0,1),L_1=L_1(a_0)\\ A_1|L_1,A_0,X\sim Bernoulli\left(expit(0.2-0.2A_0+L_1^{1/3})\right)\\ Y(a_0,a_1)\sim N(-2+2.5a_0+3.5a_1+0.5a_0{a}_1-3\exp (-L_1(a_0)),sd=1).\end{array}$$

Figure 8

RMSE and MAE of ATE Estimates Using Different Methods Under a SMART Nonlinear Model Setting.

Setting 2: SMART Linear Model, Unmeasured Covariate

Following a setting used by Daniel et al,¹⁴ this simulation considered an unmeasured confounder, U₀ (Figure 9). Specifically, the data are simulated according to the following setup:

$$\begin{array}{c}{U}_0\sim Bernoulli(0.4),\\ A_0\sim Bernoulli(0.5),\\ L_1(a_0)\sim Bernoulli\left(expit(0.25+0.3a_0-0.2U_0-0.05a_0{U}_0)\right)\\ L_1=A_0{L}_1(1)+(1-A_0)L_1(0)\\ A_1\sim Bernoulli\left(expit(0.4+0.5A_0-0.3L_1-0.4A_0{L}_1(a_0))\right)\\ Y(a_0,a_1)\sim N(0.25-0.5a_0-0.75a_1+0.2a_0{a}_1-U_0,0.2).\end{array}$$

Figure 9

RMSE and MAE of ATE Estimates Using Different Methods Under a SMART Linear Model, Unmeasured Covariate Setting.

Setting 3: SMART Kang and Schafer Dual Misspecification

We extended the well-known Kang and Schafer¹¹ simulation to a 2-stage setting (Figure 10). Like the SMART, the first treatment is assigned at random, and the outcome is a linear function of the baseline covariates Z₁ − Z₄.

$$\begin{array}{c}{Z}_1,Z_2,Z_3,Z_4\sim N(0,1),A_0\sim Bernoulli(0.5)\\ L_1(a_0)\sim N(-1.5+27.4Z_1+13.7Z_2+0\ast Z_3+0\ast Z_4-3a_0,1)\\ L_1=A_0{L}_1(1)+(1-A_0)L_1(0)\\ A_1\sim Bernoulli\left(expit(0.25A_0+(0.1)\ast L_1^{\frac{1}{3}}+0.75\ast (Z_1-0.5Z_2+0.25Z_3+0.1Z_4))\right)\\ Y(a_0,a_1)\sim N(210+L_1(a_0)+13.7Z_3+13.7Z_4-5a_0-3a_1-2a_0{a}_1,1).\end{array}$$

Figure 10

RMSE and MAE of ATE Estimates Using Different Methods Under a SMART Kang and Schafer Dual-Misspecification Setting.

As in Kang and Schafer,¹¹ only transformed covariates are observed $x_1=exp(z_1/2)$, $x_2=z_2/(1+exp(z_1))+10$, $x_3={(\frac{z_1{z}_3}{25}+0.6)}^3$, and x₄ = (z₂ + z₄ + 20)².

Setting 4: HTE SMART

It is expected that the outcome at the first stage may modify the effect of the next-stage treatment effect (Figure 11). Here, we consider a simple additive interaction effect:

$$\begin{array}{c}X\sim Bernoulli(0.4),A_0\sim Bernoulli(0.5),L_1(a_0)\sim N(0,1)\\ A_1|L_1,A_0,X\sim Bernoulli\left(expit(0.2-0.2A_0+L_1^{1/3})\right)\\ Y(a_0,a_1)|X\sim N(-2+2.5a_0+3.5a_1+0.5a_0{a}_1-3\exp (-L_1(a_0))+a_1{L}_1(a_0),sd=1).\end{array}$$

Figure 11

RMSE and MAE of ATE Estimates Using Different Methods Under an HTE SMART Setting.

Setting 5: Observational Study Adaptive Treatment Subgroup Treatment Effect

This simulation implemented a modified simulation design setting used in Schulte et al (see Figure 12).⁴³ It considered 3-level categorical baseline covariates X = −5,0,5, with multinomial distribution, $X\sim Multinomial(\frac{1}{3},\frac{1}{3},\frac{1}{3})$. The causal treatment effect at both stages varied by the baseline covariates.

$$\begin{array}{c}{A}_0\sim Bernoulli\left(expit(0.3-0.05X)\right)\\ L_1(a_0)\sim N(0.75X-0.75a_0-0.25a_{0}X,1)\\ L_1=A_0{L}_1(1)+(1-A_0)L_1(0)\\ A_1\sim Bernoulli\left(expit(0.05X+0.2A_0-0.05L_1-0.1L_1\ast A_0-0.01L_1^2)\right)\\ Y(a_0,a_1)|L_1(a_0)\sim N(3+0.5a_0+0.4a_{0}X-L_1(a_0)-L_1{(a_0)}^2+2a_1-a_0{a}_1+a_1{L}_1(a_0),1).\end{array}$$

Figure 12

RMSE and MAE of ATE Estimates Using Different Methods Under an Observational Study Adaptive Treatment Subgroup Treatment Effect Setting.

The simulation compared the 2-stage BART and GPMatch against existing causal inference methods reviewed in a recent article by Newsome et al,² including a history-adjusted structural nested model, g-computation formula, and g-estimation.

The simulation results are summarized for each of the 7 causal treatment effects of ATS in both the first and second stages. All simulation results are summarized over 200 replicates. The root mean square error (RMSE) and median absolute error (MAE) are summarized over all replicates. For GPMatch and BART, the histograms of the posterior estimates are plotted.

The simulation results suggested BART and GPMatch performed consistently better than other ATS methods, and GPMatch performed better than BART under the nonlinear model setting. Additional results for ATS are reported in Appendix C.

Developing an R Shiny Online Application

The PCORI methodology standard recommends the use of statistical causal inference methods for conducting CER. Most statistical causal inference software is created by implementing PS methods or matching methods. Software packages such as R (R Foundation for Statistical Computing), Stata (StataCorp), and SAS (SAS Institute) require steep learning curves, making them less accessible to the general research community. To our knowledge, no existing software can handle complex types of treatment, such as ATS. Therefore, one of the aims of this project was to develop an online application using R Shiny, implementing the advanced bayesian nonparametric causal inference methods of GPMatch and BART. R Shiny consists of a suite of tools for creating R online applications.¹³ For this online application, we built a GUI at the front end, which allows users to access the full function of the statistics computational engine without requiring any knowledge of the R programming language. Our online application (https://pcats.research.cchmc.org/) is open to the public and can be accessed via a web browser without installation.

CER in Patients With pcJIA

Study Design

This observational CER study used a DMARD-naive inception cohort to provide real-world evidence of the comparative effectiveness of the early-combination vs step-up CTPs in treating children with pcJIA, as recommended by the Childhood Arthritis and Rheumatology Research Alliance (CARRA).⁴ The study inclusion and exclusion criteria (Table 1) and data elements (Table 2) were designed to closely follow the CARRA recommendations. Study design details are reported in Appendix A.

Table 1

Inclusion and Exclusion Criteria for CER in Patients With pcJIA.

Table 2

Data Elements for CER in Patients With pcJIA.

Data sources

The primary data source was EMRs from the Cincinnati Children's Hospital Medical Center (CCHMC), collected in routine clinical care at a pediatric rheumatology clinic between January 1, 2009, and December 31, 2017. The secondary data source was an NIH-funded prospective follow-up study that collected detailed information on these patients and their families, such as patient-reported HRQOL, Child Health Assessment Questionnaire (CHAQ) score, and other patient-reported outcomes (PROs).¹⁰ Most of the participants in this NIH-funded research study were also being cared for at the CCHMC during the same time period. Thus, the CCHMC subset of the secondary data source was used (1) as quality control data to ensure the data quality extracted from the EMR, (2) to augment PRO data, and (3) to offer data used for sensitivity analyses.

Intervention and comparator treatment

The CTPs for children with newly diagnosed pcJIA involve 3 time-varying adaptive treatment plans. The treatments at the first stage were identified based on DMARD prescriptions (nbDMARD, bDMARD, b+nbDMARD) at baseline (ie, at the time of pcJIA diagnosis). The treatment at the second stage was identified at the 6-month follow-up visit. Following the CTP, the treatment decisions at the 3- and 9-month follow-ups were also evaluated. Because most patients continue with their previous treatment assignment, the CER study considered 2-staged ATS, first at the initial diagnosis and then at the 6-month follow-up. Patients treated with the biologic-only plan were clearly older and more likely to have large joints affected; therefore, they might represent a somewhat different patient population. To maximize equipoise and verisimilitude between the compared groups, the CER study compared the early-combination vs the step-up CTP only. Patients on the biologic-only plan were excluded. The step-up CTP is currently the most commonly adopted approach. It is expected that patients who receive the early-combination plan are likely to find matching patients who receive the step-up plan. Thus, the step-up CTP was chosen as the comparator group.

Primary and secondary outcomes

The primary outcome was cJADAS10 at the 6- and 12-month follow-up visits. The cJADAS10 is a summary score derived from the physician global assessment of disease activity (range, 0-10), patient/parent global assessment of well-being (range, 0-10), and AJC truncated at 10 (Consolaro et al⁴⁴), reflecting different perspectives of disease activity recorded in routine clinic care. The cJADAS10 is bounded between 0 and 30, with a higher score indicating more disease activity. The secondary outcome, HRQOL at the 12-month follow-up, was assessed using the Pediatric Quality of Life Inventory (PedsQL) generic module. The PedsQL generic total score is bounded between 0 and 100, with a higher score indicating better quality of life (QOL).⁴⁵

Study conduct: modification from the proposed research plan

In the original research proposal, we planned to use both multicenter registry data and single-center EMR data for the CER study. However, due to a large amount of missing data in the multicenter registry data, we were only able to conduct the CER study using the single-center EMR data. Great efforts were spent to ensure the quality of the data extracted from the EMR.

In the original proposed aim 2, the primary outcome was the American College of Rheumatology-90 (ACR90) criteria. The ACR90 is a clinical trials outcome measure calculation based on 6 core disease activity variables: (1) ESR, a blood test measure of inflammation; (2) number of joints with LOM; (3) number of joints with swelling; (4) physician's global assessment of disease activity; (5) patient/parent's global assessment of patient overall well-being activity; and (6) CHAQ score. At the time of the proposal, it was widely accepted as the best study outcome for clinical trials. However, 2 factors compelled us to make the revision:

• The CHAQ can take up to 10 minutes to complete, and there is a wide range of variation in child and parental proxy reports of function, so it is less likely to be collected at every clinic visit. In the existing data collected from real-world clinical encounters, the CHAQ score was almost never collected. As a result, ACR90 outcome could not be computed for most patients.
• The JADAS is a new disease activity measure that has been recently recommended, validated, and widely adopted as the clinical outcome measure in JIA.^44,46-49 This composition measure summarizes across 4 core disease activity measures: physician's (1) global assessment of disease activities, (2) patient/parent's global assessment of patient overall well-being, (3) number of joints with swelling, and (4) ESR. The JADAS measures ongoing disease activity at a single point in time and allows a comparison of disease activity between patients.^46,50 The cJADAS10, with a cutoff at 10 for active joint count, is a clinical abbreviated version of the JADAS, which has been recommended and used in studies implementing observational data.^47,48,51 After discussions with the stakeholder board and clinical collaborators, we revised the primary study outcome from the JADAS to the cJADAS10. This is the same outcome used in the PR-COIN project.

These revisions were communicated to and approved by a PCORI program officer.

PCATS Online R Shiny Application

The GPMatch and extended BART methods have been implemented in an easy-to-use publicly available online application (https://pcats.research.cchmc.org/). It allows users to upload their own data and specify outcome, treatment, confounding, and prognostic variables. The outputs are presented in a table comparing the 2 treatment arms side by side on the selected confounding and prognostic variables, and ATE estimates and predicted potential outcomes are presented in both tables and figures. If the HTE option is selected, then the HTE estimates will also be presented. A detailed user's guide has been developed (Appendix E) to provide examples and step-by-step instructions for some commonly encountered CER problem settings, including continuous, multilevel, categorical, and mixed composite types of treatment, either adaptive or nonadaptive. These examples facilitate better user experiences. The process flow for the PCATS application is presented in Figure 13.

Figure 13

Process Flow for the PCATS Online Application.

Overview of the PCATS Application

In the upper-left corner of the PCATS online application (Figure 14), users have an option to login using Gmail accounts, which allows them to save their projects. The PCATS application accepts data files in CSV or Excel formats. Once users upload their data (Figure 15), they can review the data, define data types of variables (numerical vs categorical), choose a treatment type (adaptive vs nonadaptive), and determine advanced parameters (ie, number of MCMC samples). The models for ATS (Figure 16B) are different from those for nonadaptive treatment strategies (Figure 16A) because users need to specify the outcome and covariates for each stage in the models. While building the model, users can also select factor(s) to test for HTEs. Once users complete building their models and click “Run Model,” the PCATS application will analyze the data using the GPMatch, extended BART, or BART methods depending on the types of outcomes and treatment. The analysis results are presented in tables and interactive figures within the online application. An example of the outputs is presented in Figure 17. For step-by-step instructions on how to use the PCATS application, please refer to Appendix E.

Figure 14

PCATS Application Webpage.

Figure 15

Review Data Using the PCATS Application.

Figure 16

Build Statistical Models Using the PCATS Application.

Figure 17

Example of Analysis Outputs Using the PCATS Application.

CER in Patients With pcJIA

Patient Eligibility Screening

Out of 1750 patients with JIA captured in the EMR, only 530 patients met the eligibility criteria of being aged 1 to 19 years, newly diagnosed, DMARD naive, and diagnosed with pcJIA. Out of these 530 eligible patients, 47 patients had at least 1 of the specified comorbid conditions (ie, celiac disease, trisomy 21, and inflammatory bowel disease [IBD]) and were excluded. Additionally, 76 patients were excluded because they were following a biologic-only CTP. As described earlier, this is for the purpose of equipoise and verisimilitude. Detailed patient eligibility screening is summarized in Figure 18.

Figure 18

Eligibility Screening in Patients With JIA.

Primary Outcome: cJADAS10 at 6 Months and 12 Months

Results from the cJADAS10 were collected from 194 (65.55%) and 91 (73.39%) patients in the step-up and early-combination groups at baseline, respectively (Table 3). Because of retrospective EMR data collection, the number of missing cJADAS10 results was higher at follow-up visits than at baseline. Missing imputation was performed and is presented in Appendix D.

Table 3

Nonmissing cJADAS10 Results at Study Visits.

After 6 months of treatment, compared with the step-up CTP, the patients participating in the early-combination CTP demonstrated a greater reduction in disease activity. The GPMatch result suggested that both CTPs were effective in improving data activities. It predicted that the expected mean ± SD cJADAS10 scores at 6 months would be 6.7 ± 0.48 points and 4.7 ± 0.66 points for the step-up and early-combination approaches, respectively. The early-combination CTP led to a significantly greater reduction in cJADAS10 scores, at −1.98 (95% CI, −3.55 to −0.40).

The causal inference analyses estimate the potential outcomes for each of the 4 ATS. Depending on the disease progression at the end of the first-line treatment, the medication might be adjusted, and then the cJADAS10 outcome was measured at the end of the second-line treatment. After 6 months of treatment, patients on the step-up CTP might escalate treatment (ie, escalate their initial nbDMARD, change to a different nbDMARD, or add a bDMARD); alternatively, they might de-escalate/maintain initial treatment (ie, stop the DMARD or continue taking the same prescription). Patients on the early-combination CTP might escalate/maintain the initial treatment approach (ie, change DMARDs or continue taking the same combination of DMARDs), or they might de-escalate treatment (ie, stop the bDMARD and/or nbDMARD). The GPMatch method estimated the posterior distribution of cJADAS10 outcome and also whether the patient had demonstrated improvement at each ATS. The GPMatch-predicted results (mean [95% CI]) of potential cJADAS10 outcomes are presented in Figure 19.

Figure 19

Estimated cJADAS10 Outcome Using the GPMatch Method.

Overall, the results suggested both CTPs were effective in improving disease activity (Figure 19). Early-combination treatment, on average, produced a significant 2-point reduction in the cJADAS10, with an ATE of −1.98 (95% CI, −3.55 to −0.40) by 6 months, which was sustained up to 12 months. The study did not identify HTE by JIA subtypes or baseline cJADAS10.

Summary of the Project

This project is motivated by the need to evaluate the comparative effectiveness of ATS used routinely in treating patients with chronic or prolonged disease conditions. We recognized some important limitations and a need to improve statistical causal inference methods for ATS. These limitations may threaten the validity of the CER results, which could lead to inconsistent, confusing, or even misinformed conclusions.

The results of this project offered some important improvements to statistical causal inference methods. First, we have developed a novel full bayesian DR causal inference method, GPMatch.³⁶ This method can be considered an extension to the bayesian marginal structural model. By using GP covariance functions as a matching tool, GPMatch combines the virtue of matching and bayesian nonparametric modeling within a single step of the full bayesian framework. Requiring relatively weaker causal assumptions, the GPMatch method produces DR average causal treatment effects, enjoys well-calibrated frequentist properties, and outperforms many existing causal inference methods under the most realistic dual-misspecification setting (ie, neither knowledge of the true outcome-generating process nor treatment selection is known). Appendices B and C provide detailed results.

Second, the widely adopted 3 causal assumptions initially proposed for the theory of PS are widely acknowledged as overly strong.^52,53 Alternatively, DAGs¹⁵ and other frameworks have been proposed, such as in Iacus et al.¹⁶ Here, we presented the causal assumptions with a DAG within a counterfactual world setting. The DAG explicitly acknowledges 4 sources of potential unmeasured confounders and therefore presents a set of relatively weaker versions of causal assumptions that are more suitable for real-world data. For example, we relaxed the SUTVA to SUTVEA. Instead of requiring the observation to be an exact copy of the corresponding potential outcomes, that is, $Y_i=A_i{Y}_i^{(1)}+(1-A_i)Y_i^{(0)}$, GPMatch allows a noisy version of the potential outcome, that is, $Y_i=A_i{Y}_i^{(1)}+(1-A_i)Y_i^{(0)}+{\epsilon }_i$. When it comes to real-world data, it is almost always true that the outcome measures are subject to measurement error, and that the treatment effect could differ by some unobserved factors, such as presurgery procedures, timing and seasonal effects, concomitant medications, and food or drink taken together with the medications. Relaxing the causal assumptions and explicitly acknowledging the unmeasured source of random effect helps ensure the validity of the results.

Third, we have created a GUI application, PCATS (https://pcats.research.cchmc.org/), implementing the GPMatch and BART methods for CER. This R Shiny application allows users to upload their data, provides step-by-step GUI instructions, and reports tables and figures of estimates of ATE and the predicted potential outcomes. The figures are interactive, allowing users to hover their cursor over any point of interest, and the application will provide the corresponding probability estimates. The application is open to the general public, offering solutions to some important challenges in CER: (1) complex types of treatment schedules such as ordinal, mixed types of categorical and ordinal, and time-varying adaptive or nonadaptive treatment; (2) conditional causal treatment effects; and (3) bounded summary score outcomes, such as patient-reported HRQOL and many other patient-reported and clinical summary score outcomes. With the PCATS app, researchers can perform CER using state-of-the-art bayesian causal inference methods without the need to learn R or other data analytic languages. Therefore, it facilitates more timely, rigorous, and reproducible CER.

Last, the second aim of this project evaluated the effectiveness of the early-combination CTP in children with newly diagnosed pcJIA, compared with the more conventional approach of the step-up CTP, on clinical and QOL outcomes at 6 and 12 months of treatment. To our knowledge, this is the first study that applies causal inference methods to evaluate the comparative effectiveness of early-combination vs step-up CTPs for patients with pcJIA using EMR data.

As a full bayesian method, GPMatch is highly flexible and versatile and can be extended to consider many additional challenges inherent to analyzing real-world data. The online application makes available these advanced bayesian causal inference methods that are directly applicable to CER. In conclusion, the project has made a significant contribution to methods improvement for CER, and it has the potential to contribute significantly to the advancement of evidence-based health care.

GPMatch Method

The proposed GPMatch method offers a full bayesian causal inference approach that can effectively address the unique challenges inherent in causal inference. First, using GP prior covariance function to model the covariance of the observed data, GPMatch can estimate missing potential outcomes much like matching methods do, while avoiding the pitfalls of many matching methods. No data are discarded, and no arbitrary caliper is required. Instead, the model allows empirical estimation of the length-scale and variance parameters. The squared-exponential covariance function of GP prior offers an alternative distance metric, which closely resembles MD. It matches individuals by the degree of matching proportional to the squared-exponential distance without requiring specification of caliper. For this reason, GPMatch can use data more effectively than usual matching procedures can. Different length-scale parameters are considered for different covariates used in defining squared-exponential covariance function. This allows the data to select the most important covariates to be matched on and acknowledges that some variables are more important than others. Although the idea of using GP prior for bayesian causal inference is not new, using the GP covariance function as a matching device is a unique contribution of this study. The matching utility of GP covariance function is presented analytically by considering a setting when the matching structure is known. We show that GPMatch enjoys DR properties in the sense that it correctly estimates the ATE when 1 of the following conditions is true: (1) The mean function of the GPMatch correctly specifies the prognostic function of the potential outcome; or (2) the GP prior covariance function correctly specifies matching structure. We show that GPMatch estimates the treatment effect by inducing independence between 2 residuals: the residual from the treatment propensity estimate and the residual from the outcome estimate, much like the g-estimation method. Unlike the 2-staged g-estimation, the estimations of the parameters in covariance function and the mean function for the GPMatch are performed simultaneously. Therefore, the GPMatch regression approach, which can integrate the benefits of the regression model and matching method, offers a natural way for bayesian causal inference to address challenges unique to causal inference. The robust and efficient proprieties of GPMatch are well supported by the simulation results designed to reflect the most realistic settings (ie, no knowledge of the matching or functional form of the outcome model is available).³⁶

CER in Patients With pcJIA

In this study, the comparative effectiveness of the early-combination CTP in children with newly diagnosed pcJIA was compared with the more conventional approach of the step-up CTP on clinical and QOL outcomes after 6 and 12 months of treatment. To our knowledge, this is the first study that applies causal inference methods to evaluate comparative effectiveness of early-combination vs step-up CTP using EMR data. Within an established EMR system, such interactions could be tracked from the first date of diagnosis throughout the course of disease progression and treatment, particularly for patients with chronic conditions. Therefore, it is an invaluable data source for evaluating the effectiveness of alternative treatment choices, understanding potential HTE, and subsequently guiding evidence-based treatment decisions. This study demonstrates that the EMR could be used for better understanding treatment effectiveness.

Method Development

GPMatch offers robust, accurate, and efficient estimation of ATE. As a full bayesian approach, it is flexible and general and can be extended to address more complex data types and structures inherent in real-world settings. In this study, the model has been extended to a time-varying ATS setting, with consideration of time-dependent confounding. An online application implemented BART and GPMatch for both adaptive and nonadaptive treatments, making the advanced bayesian causal inference method directly accessible to the general research community. Further development of the GPMatch method and the application can address additional challenges, such as multilevel or cluster data structure, missing data, and HTE.

CER in Patients With pcJIA

This observational study offers the first real-world evidence of CTPs. We found that both early-combination and step-up CTPs are effective in reducing disease activity, and that early-combination CTP is more effective, leading to better disease activity scores (ie, cJADAS10) after 6 and 12 months of treatment. This finding signifies the need for future studies to investigate the comparative effectiveness of CTPs on long-term outcomes such as cJADAS10, inactive disease, and HRQOL.

Appendix A.

PCATS Study Protocol (PDF, 1.0M)

Appendix B

PCATS Aim 1a: GPMatch Methodology Manuscript (PDF, 905K)

Appendix C.

PCATS Aim 1b: 2-Stage Treatment Assignment Results (PDF, 3.8M)

Appendix D.

PCATS Aim 2: CER Report on CTPs for Patients With pcJIA (PDF, 1.4M)

Appendix E.

PCATS Online Application User Guide (PDF, 2.0M)