Review and Appraisal of Indirect Treatment Comparisons
Review of Manufacturer-Provided Network Meta-Analysis37
Objectives and Rationale for Manufacturer’s Network Meta-Analysis
The objective of this report was to evaluate the comparative efficacy, safety, and health-related quality of life (HRQoL) impact of crisaborole 2% ointment versus other topical pharmacologic therapies, for the treatment of mild-to-moderate AD, using an NMA approach.
Methods for Manufacturer’s Network Meta-Analysis
Study Eligibility and Selection Process
The authors indicated that a systematic review and associated NMA were conducted according to the requirements of major health technology assessment agencies such as the National Institute for Health and Care Excellence (NICE) and the Cochrane Collaboration and reported according to the general guidelines described by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) Statement.
The NMA was based on the systematic review of the literature which included both electronic and manual search components. Multiple databases were searched from inception to September 1, 2017. The search was restricted to articles published in English.
It is unclear whether the selection criteria were defined a priori. The main inclusion criteria for the systematic review were randomized controlled trials (RCTs) that recruited children (two years and older) and adult patients with a clinical diagnosis of mild-to-moderate AD. Only topical therapies were included. To be eligible, the studies were required to report at least one of the following outcomes: change in AD severity, HRQoL, or safety. Study selection was accomplished through two levels of screening by two independent researchers. Any disagreements were resolved through discussion and consensus.
Data Extraction
Data were extracted by one reviewer, and verified by a second reviewer. Any disagreements were resolved by consensus.
Comparators
Topical pharmacological therapies such as TCS and TCI were of interest for inclusion in the NMA.
Outcomes
The main end points of interest included in the systematic review were stated to be:
AD severity: measured with Investigator’s Static Global Assessment (ISGA) scores (or other AD severity scales), proportion of patients with greater than and equal to 2-grade improvement in ISGA (or similar scale) to clear (0) or almost clear (1), at the time points of day 7 to 8; day 14 to 15; day 21 to 22; day 28 to 29; and day 42 to 43.
HRQoL: measured with Short Form (36) Health Survey or EuroQol 5-Dimensions questionnaire.
Safety: overall adverse events (AEs), discontinuation due to AEs.
Quality Assessment of Included Studies
All included RCTs were evaluated for risk of bias using the Cochrane Risk of Bias tool for RCTs, which summarizes how well each study meets the following five quality criteria: study randomization, concealment of treatment allocation, missing outcome data, blinding of outcome measurement, and completeness of reporting of outcomes, with an overall quality score awarded to each study. There was no description on how the results of risk of bias assessment could have an impact on data analysis.
Indirect Comparison Methods
All analyses were conducted within a Bayesian framework using OpenBUGS. All analyses involved a 50,000 run-in iteration phase and a 50,000 iteration phase for parameter estimation. All calculations were performed using OpenBUGS 3.2.2.
The primary end point of interest of this NMA was ISGA score of 0 to 1 at 28 to 29 days. For the ISGA response outcome as well as AEs, there were notable differences across the studies in the size of the baseline risk such as the likelihood of response on vehicle. Given the need to consider baseline risk, a class-effects model was implemented. The authors indicated that such a model would allow for a more stable estimation of the beta (the effect [slope] for baseline risk on relative effects). Specifically, all non-vehicle treatments were grouped into one class, and vehicle into its own class of treatment, with an assumption that non-vehicle treatments varied randomly in efficacy, around a common mean. Fixed or random-effects models (or both, where appropriate) were used for the analyses: class-effects models with baseline risk were fixed-effects for treatment and random-effects for treatments within class, in which all non-vehicle treatments were considered to share a class.
In the analyses, model fit was explored by comparing the deviance information criterion and the posterior mean of the residual deviance for the fixed-effect and random-effect models. Convergence (which is required in Bayesian models in order for the estimates to be valid) was confirmed by evaluating the three-chain Brooks-Gelman-Rubin plots and inspection of the ratios of Monte Carlo error to the standard deviations (SDs) of the posteriors. If convergence was not achieved, then the run-in was increased and/or other factors were examined (such as choice of prior and starting values).
Four analytic models were developed in this study:
Analysis A: Unadjusted for baseline risk; both random-effect and fixed-effect models were employed; random-effect model used a prior of U[0,1] for the SD of treatment effects across studies within treatment comparisons.
Analysis B: Baseline risk adjustment was performed only with default prior; only fixed-effect model was used (due to convergence issues); the random-effect model was not run.
Analysis C: Baseline risk with default prior + class-effects adjustment, prior of U[0,1] for the SD of treatment effects within class; both random-effect and fixed-effect models were employed; an assumption was that all non-standard of care (SOC) treatments share a common relative effect versus SOC, and vary randomly around that effect.
Analysis D: Baseline risk with default prior + class-effects adjustment, prior of U[0, 0.5] for the SD of treatment effect within class; both random-effect and fixed-effect models were employed; an assumption was that all non-SOC treatments share a common relative effect versus SOC, and vary randomly around that effect.
None of the HRQoL outcomes had sufficient data identified in the systematic literature review for NMA to be feasible. In addition, due to a lack of data, subgroup analyses by age and disease severity were not feasible.
Results
In total, nine RCTs of patients with AD were identified for the NMA, including the AD-301 and AD-302 trials. Treatment durations of these nine RCTs ranged from 28 days to 42 days, and the follow-up duration ranged from 21 days to 43 days. The sample size of these RCTs varied between 133 and 764. Among them, all the included RCTs examined patients older than two years of age, except for one study which enrolled a patient population of one year to 17 years of age and reported an overall average patient age of 6.7 years. One study enrolled patients older than 16 years (average 39.1). Most of the trials (44%) reported on pediatric populations (age range from two years to 17 years), with an additional 33% of studies reporting on a combination of both adult and pediatric patients. For one of the RCTs, the mean age was not reported. The average age of patients in the included RCTs ranged from 6.4 years to 39.1 years. Six RCTs enrolled mixed mild-to-moderate AD populations, while one trial enrolled exclusively mild patients and two trials enrolled exclusively moderate patients.
The interventions in trials of AD were: crisaborole 2% ointment, TCI (pimecrolimus 1% cream; tacrolimus 0.03% or 0.1% ointment), and vehicle. The vehicles used in the included trials were formulated with different emollient properties. None of the included trials reported on the contents of the vehicle, or the proportion of the ingredients. The contents of the study drugs were identified from clinical study reports or drug labels. The contents of the vehicles were assumed to be identical to the contents of the base used for the interventions (white petrolatum, propylene glycol, monoglycerides, diglycerides, paraffin, triglycerides, mineral oil, etc.) by the authors.
The authors reported that the overall risk of bias of the included RCTs was low. The authors indicated that the main reasons for the poor quality of the trials were inadequate treatment allocation concealment and incompleteness of reporting of outcomes. The AD-301 and AD-302 trials were generally considered at low risk of bias.
▬▬▬▬▬
Atopic Dermatitis Severity
▬▬▬▬▬
▬▬▬▬▬ 1%, or between crisaborole 2% and tacrolimus 0.03% with respect to achieving an ISGA score of 0 to 1 at 28 to 29 days ().
▬▬▬▬▬
Table 20Efficacy Outcome Measures in the Manufacturer’s Network Meta-Analysis
View in own window
| Analysis B (Fixed-Effect Model) | Analysis C (Random-Effect Model) |
|---|
| ISGA Score of 0 to 1 at 28 to 29 Days, OR (95% Crl) |
|---|
| Crisaborole 2% vs. pimecrolimus 1% | ▬ | ▬ |
|---|
| Crisaborole 2% vs. tacrolimus 0.03% | ▬ | ▬ |
|---|
| ISGA Score of 0 to 1 at 7 to 8 Days, OR (95% Crl) |
|---|
| Crisaborole 2% vs. pimecrolimus 1% | ▬ | ▬ |
|---|
| Crisaborole 2% vs. tacrolimus 0.03% | ▬ | ▬ |
|---|
| Crisaborole 2% vs. tacrolimus 0.1% | ▬ | ▬ |
|---|
| ISGA Score of 0 to 1 at 14 to 15 Days, OR (95% Crl) |
|---|
| Crisaborole 2% vs. pimecrolimus 1% | ▬ | ▬ |
|---|
| Crisaborole 2% vs. tacrolimus 0.03% | ▬ | ▬ |
|---|
| ISGA Score of 0 to 1 at 21 to 22 Days, OR (95% Crl) |
|---|
| Crisaborole 2% vs. pimecrolimus 1% | ▬ | ▬ |
|---|
| Crisaborole 2% vs. tacrolimus 0.03% | ▬ | ▬ |
|---|
| Crisaborole 2% vs. tacrolimus 0.1% | ▬ | ▬ |
|---|
| ISGA Score of 0 to 1 at 28 to 43 Days, OR (95% Crl) |
|---|
| Crisaborole 2% vs. pimecrolimus 1% | ▬ | ▬ |
|---|
| Crisaborole 2% vs. tacrolimus 0.03% | ▬ | ▬ |
|---|
| Crisaborole 2% vs. tacrolimus 0.1% | ▬ | ▬ |
|---|
CrI = credible interval; ISGA = Investigator’s Static Global Assessment; OR = odds ratio.
Median values of the point estimates of the ISGA score were presented.
Source: Manufacturer-provided NMA.37
Safety
The authors indicated that an NMA of safety outcomes was not performed because of considerable heterogeneity related to:
- 1)
Inconsistent reporting of data for comparators among included trials and variation in outcome definitions across included trials;
- 2)
It was unclear if an outcome was not reported if it was due to the threshold or definitions or the outcome simply did not occur;
- 3)
Differences in study period between trials (changes in reporting of outcomes data over time; older versus newer trials); and
- 4)
The sparsity of data among included trials.
Therefore, safety results were qualitatively described in the manufacturer-provided NMA.
The most common AE associated with treatment with crisaborole 2% ointment was application site pain, such as burning or stinging. A less common (less than 1%) AE in patients treated with crisaborole 2% ointment included contact urticaria. The use of TCI was related to local symptoms, such as skin burning (burning sensation, stinging, soreness) or pruritus.
Health-Related Quality of Life
An NMA of HRQoL outcomes was not performed due to the insufficient data identified in the systematic review.
Critical Appraisal
In the manufacturer-provided NMA, the analyses were based on a systematic review of the literature to identify all relevant studies. The literature search included only English-language articles. The methods for study selection and data extraction were suitable. Risk of bias of all individual studies was assessed using the Cochrane Risk of Bias tool, and was generally low as reported by the authors. However, inadequate treatment allocation concealment and incompleteness of reporting of outcomes, as indicated by the authors as the main reasons, could be deemed as a high risk of bias, leading to poor quality of the trial; there was no description of how the results of risk of bias assessment could have an impact on data analysis (e.g., excluding the poor study or conducting a sensitivity analysis, etc.).
Potential sources of heterogeneity with respect to the baseline characteristics were identified, such as age (which ranged from six years to 39 years) and disease severity (six trials enrolled mild-to-moderate patients, one enrolled exclusively mild patients, and two enrolled exclusively moderate patients). However, subgroup analyses by age or baseline disease severity were not conducted, due to the insufficient number of trials within the network. Therefore, there is also uncertainty as to whether relative treatment effects differ by patient age.
In the main report, the primary efficacy outcome in the two pivotal studies was “success in ISGA,” which was defined as ISGA of clear (0) or almost clear (1) with at least a 2-grade improvement from baseline at day 29. In the NMA, the primary efficacy outcome was achieving ISGA score of 0 to 1 at 28 to 29 days. This was a secondary end point from the two pivotal trials. Given that the secondary end point may not be powered to show statistical significance of the difference between treatment groups (type II error), this would make the findings from the NMA more difficult to interpret. If there is statistically significant difference in favour of crisaborole on this secondary end point, we cannot be sure whether observed effect is just a random effect, or whether it is simply due to heterogeneity between trials. ▬▬▬▬▬
TCS is an important treatment option for patients with AD. There is no indirect comparative evidence between crisaborole and TCS such as betamethasone, therefore we are not able to examine the relative efficacy and safety of crisaborole versus TCS in the study population. In addition, there is no indirect comparative evidence for HRQoL or safety outcomes, due to the limited number of trials or sparsity of data.
Treatment durations of the included trials were short (ranging from 28 days to 42 days). The long-term efficacy and safety of crisaborole relative to other topical pharmacological therapies are unknown.
Review of the Incremental Cost-Effectiveness Ratio Review38
Objectives and Rationale for the Incremental Cost-Effectiveness Ratio Network Meta-Analysis
To evaluate the comparative clinical effectiveness of crisaborole versus the emollient for management of mild-to-moderate AD.
Methods for the Incremental Cost-Effectiveness Ratio Network Meta-Analysis
This report was based on a review of the literature which included both electronic and manual search components. Multiple databases were searched from January 1996 to January 2017. One single reviewer screened the literature. Eligibility criteria for this study are presented in . Overall, clinical trials or high-quality systematic reviews of crisaborole or dupilumab compared with emollient or placebo were included. Clinical benefit and safety of the study drugs were assessed. In addition, findings from previously published systematic reviews to inform comparisons of crisaborole to TCS and TCI (with the exception of pimecrolimus, where an NMA was conducted for the comparison between crisaborole and pimecrolimus), and comparisons of dupilumab to cyclosporine, phototherapy, and failed topical therapies were qualitatively reviewed. The criteria published by the US Preventive Services Task Force (USPSTF) were adopted to assess the quality of RCTs and comparative cohort studies.
The findings for the comparisons of dupilumab to the comparators are not presented in this CDR review.
The authors indicated that this review was conducted in accordance with the PRISMA guidelines. The study was not sponsored by the industry.
Data Extraction
It is unclear whether data extraction was performed by two independent reviewers.
Comparators
Crisaborole was planned to compare with emollient, TCS, and TCI. An NMA was conducted to compare crisaborole with pimecrolimus.
Outcomes
Clinical benefits and harms were reported in the study.
Quality Assessment
The criteria published by the US Preventive Services Task Force (USPSTF) were adopted to assess the quality of RCTs and comparative cohort studies.
Evidence Network
Not available.
Meta-Analysis and Indirect Comparison for the Incremental Cost-Effectiveness Ratio Network Meta-Analysis
Crisaborole was evaluated in Studies AD-301 and AD-302 on a 5-point ISGA score. Two other trials (Eichenfield 2002 and Ho 2003) comparing TCI (pimecrolimus) to placebo, using a 6-point ISGA score as an end point were also identified to provide indirect evidence for the comparison between crisaborole and other active comparators. The severity of disease was similar between trials with regard to baseline ISGA score and per cent body surface area involved. Given the lack of head-to-head data and the similar versions of the ISGA score, an indirect comparisons using Bayesian approach was conducted. This analysis was conducted assuming that the clear and almost clear categories were similar on both 5-point and 6-point scales of the ISGA. Random-effect models were used for the analysis.
Results of the Incremental Cost-Effectiveness Ratio Network Meta-Analysis
Data from four trials (AD-301, AD-302, Eichenfield 2002, and Ho 2003) were included in the NMA. The first three trials were also included in the manufacturer-provided NMA. There was no statistically significant difference in efficacy found between crisaborole and pimecrolimus based on ISGA score. The risk ratio for achieving an ISGA score of 0 to 1 between crisaborole and pimecrolimus was 0.61 (95% credible interval [CrI] 0.10 to 2.28; time point not specified).
An NMA was not performed on safety outcomes for the comparison between crisaborole and other active topical therapies.
Critical Appraisal of the Incremental Cost-Effectiveness Ratio Network Meta-Analysis
A systematic review approach was not employed to identify the potentially relevant clinical trials. One single reviewer screened the abstracts and full reports. It was unclear whether data extraction and quality assessment of the included studies were performed by two reviewers. Some important patient baseline characteristics were not described, such as age, prior experience with topical therapies for AD. Subgroup analysis by important patient baseline characteristics, such as age or disease severity was not reported. Therefore, there is also uncertainty on whether relative treatment effects differ by patient age.
In this ICER study, many efficacy outcomes were qualitatively reviewed. An NMA was performed only for the comparison between crisaborole and pimecrolimus. It is unclear why other common topical therapies were not included in the analysis. Similar to the manufacturer-provided NMA, the efficacy outcome in this NMA was the proportion of patients who achieved an ISGA score of 0 to 1. This is a secondary efficacy outcome in AD-301 and AD-302; therefore may not have sufficient power to show statistical significance of the difference between treatment groups.
Similar to the manufacturer-conducted NMA, the included trials in the ICER study were performed in very different time periods (2016 versus 2002 and 2003) and used different versions of ISGA scales (5-point and 6-point). Given the considerable heterogeneity at baseline patient characteristics and the limited number of trials included in the network, it is challenging to make concrete conclusions regarding the relative efficacy of crisaborole and other active treatments.
Conclusion
There are no head-to-head trials comparing crisaborole 2% ointment to other topical pharmacological therapies for patients with mild-to-moderate AD. In the absence of direct evidence, two NMAs comparing crisaborole to other topical pharmacological therapies were identified and summarized for this review. Only one outcome, the proportion of patients achieving an ISGA score of 0 to 1, was assessed in the NMAs, and data were only available for the comparison between crisaborole and TCI (pimecrolimus and tacrolimus). ▬▬▬▬▬ However, both NMAs were limited by the number of trials available to inform the network, the fact that only comparisons versus TCI (pimecrolimus and tacrolimus) were reported, that subgroup analyses based on age were not conducted, that there was reporting of only one efficacy outcome (achieving an ISGA score of 0 to 1) to assess comparative treatment effects, and that there was no quantitative assessment of comparative safety. Due to the limitations in the analyses and uncertainty as to whether relative treatment effects differ by patient age, no definitive conclusions regarding the comparative efficacy and safety of crisaborole to other topical therapies can be made for either pediatric or adult patients with mild-to-moderate AD.
