3.3.1.1. Data synthesis for intervention reviews

Where possible, meta-analyses were conducted to combine the results of studies for each review question using Cochrane Review Manager (RevMan5) software or STATA. Fixed-effects (Mantel-Haenszel) techniques were used to calculate risk ratios (relative risk) for the binary outcomes.

For the continuous outcomes, measures of central tendency (mean) and variation (standard deviation) were required for meta‐analysis. A generic inverse variance option in RevMan5 was used if any studies reported solely the summary statistics and 95% confidence interval (95% CI) or standard error; this included any hazard ratios reported. However, in cases where standard deviations (SDs) were not reported per intervention group, the standard error (SE) for the mean difference was calculated from other reported statistics (probability [p] values or 95% CIs) if available: meta‐analysis was then undertaken for the mean difference and SE using the generic inverse variance method in RevMan5. When the only evidence was based on studies that summarised results by presenting medians (and interquartile ranges), or only p values were given, this information was assessed in terms of the study’s sample size and was included in the GRADE tables as a narrative summary. Consequently, aspects of quality assessment such as imprecision of effect could not be assessed for this evidence and this has been recorded in the footnotes of the GRADE tables.

In instances where multiple scales were reported for a single outcome, mean differences were standardised (divided by their SD) before pooling, giving meta-analysed results that were reported as standardised mean differences (SMD), with a standard deviation of 1.

Where reported, time-to-event data were presented as a hazard ratio or results from a Cox hazard proportion model were given as a result from a multivariate analysis.

Stratified analyses were predefined for some review questions at the protocol stage when the Committee identified these strata to be different in terms of clinical characteristics and the interventions were expected to have a different effect, for example on the management of short-term symptoms. We stratified our analysis for women with a uterus, women without a uterus and women with a history of or at risk of breast cancer. Statistical heterogeneity was assessed by visually examining the forest plots, and by considering the chi-squared test for significance at p<0.1 or an I-squared inconsistency statistic (with an I-squared value of 50 to 74.99% indicating serious inconsistency and I-squared value of over 75% indicating very serious inconsistency). If the heterogeneity still remained, a random effects (DerSimonian and Laird) model was employed to provide a more conservative estimate of the effect. For meta-analyses with serious heterogeneity, but no pre-defined strata for stratified analysis, basic sensitivity analyses on features such as age, gender and study types were carried out.

3.3.1.2. Data synthesis for diagnostic test accuracy review

For diagnostic test accuracy studies, the following outcomes were reported:

• sensitivity
• specificity
• positive and negative likelihood ratio
• area under the curve (AUC).

3.3.1.3. Data synthesis for qualitative review

For the qualitative review in the guideline, results were reported narratively either by individual study or by summarising the range of values as reported across similar studies, following basic thematic analysis. A summary evidence table was used when data allowed for this.

3.3.3.1. Grading the quality of clinical evidence

After results were pooled, the overall quality of evidence for each outcome was considered. The following procedure was adopted when using the GRADE approach:

• A quality rating was assigned based on the study design. RCTs start as high, observational studies as low and uncontrolled case series as low or very low.
• The rating was then downgraded for the specified criteria: risk of bias (study limitations); inconsistency; indirectness; imprecision; and publication bias. These criteria are detailed below. Evidence from observational studies (which had not previously been downgraded) was upgraded if there was a large magnitude of effect or a dose-response gradient, and if all plausible confounding would reduce a demonstrated effect or suggest a spurious effect when results showed no effect. Each quality element considered to have ‘serious’ or ‘very serious’ risk of bias was rated down by 1 or 2 points, respectively.
• The downgraded and upgraded ratings were then summed and the overall quality rating was revised. For example, all RCTs started as high and the overall quality became moderate, low or very low if 1, 2 or 3 points were deducted respectively.
• The reasons or criteria used for downgrading were specified in the footnotes.

The details of the criteria used for each of the main quality elements are discussed further in Sections 3.3.3.2 to 3.3.3.6.

3.3.3.2. Risk of bias

Bias can be defined as anything that causes a consistent deviation from the truth. Bias can be perceived as a systematic error; for example, if a study was carried out several times and there was a consistently wrong answer, the results would be inaccurate.

The risk of bias for a given study and outcome is associated with the risk of over‐ or underestimation of the true effect.

The risks of bias are listed in Table 8.

Table 8

Risk of bias in randomised controlled trials.

A study with a poor methodological design does not automatically imply high risk of bias; the bias is considered individually for each outcome and it is assessed whether this poor design will impact on the estimation of the intervention effect.

3.3.3.3. Diagnostic studies

For diagnostic accuracy studies, the Quality Assessment of Diagnostic Accuracy Studies version 2 (QUADAS‐2) checklist was used. Risk of bias and applicability in primary diagnostic accuracy studies in QUADAS‐2 consists of 4 domains (see Figure 3):

• patient selection
• index test
• reference standard
• flow and timing.

Figure 3

Summary of QUADAS-2 with a reference to quality domains.

3.3.3.4. Inconsistency

Inconsistency refers to an unexplained heterogeneity of results. When estimates of the treatment effect across studies differ widely (that is, when there is heterogeneity or variability in results), this suggests true differences in underlying treatment effect.

Heterogeneity in meta‐analyses was examined and sensitivity and subgroup analyses performed as pre‐specified in the protocols (Appendix D).

When heterogeneity existed (chi-squared p less than 0.1, I-squared inconsistency statistic of between 50% and 74.99% or I-squared greater than 50% or evidence from examining forest plots), but no plausible explanation was found (for example, duration of intervention or different follow-up periods) the quality of evidence was downgraded by 1 or 2 levels, depending on the extent of uncertainty to the results contributed by the inconsistency in the results. In addition to the I-squared and chi-squared values, the decision for downgrading was also dependent on factors such as whether the intervention is associated with benefit in all other outcomes or whether the uncertainty about the magnitude of benefit (or harm) of the outcome showing heterogeneity would influence the overall judgment about net benefit or harm (across all outcomes).

When outcomes are derived from a single trial, inconsistency is not an issue for downgrading the quality of evidence. However, ‘no inconsistency’ is nevertheless used to describe this quality assessment in the GRADE tables.

3.3.3.5. Indirectness

Directness refers to the extent to which the populations, intervention, comparisons and outcome measures are similar to those defined in the inclusion criteria for the reviews. Indirectness is important when these differences are expected to contribute to a difference in effect size or may affect the balance of harms and benefits considered for an intervention.

3.3.3.6. Imprecision

Imprecision in guideline development concerns whether the uncertainty (confidence interval) around the effect estimate means that it is not clear whether there is a clinically important difference between interventions or not. Therefore, imprecision differs from the other aspects of evidence quality in that it is not really concerned with whether the point estimate is accurate or correct (has internal or external validity) but instead is concerned with the uncertainty about what the point estimate is. This uncertainty is reflected in the width of the confidence interval.

The 95% confidence interval (95% CI) is defined as the range of values that contain the population value with 95% probability. The larger the trial, the smaller the 95% CI and the more certain the effect estimate.

Imprecision in the evidence reviews was assessed by considering whether the width of the 95% CI of the effect estimate was relevant to decision‐making, considering each outcome in isolation.

When the confidence interval of the effect estimate is wholly contained in 1 of the 3 zones (clinically important benefit, clinically important harm, no clinically important benefit or harm) we are not uncertain about the size and direction of effect (whether there is a clinically important benefit, or the effect is not clinically important, or there is a clinically important harm), so there is no imprecision.

When a wide confidence interval lies partly in each of 2 zones, it is uncertain in which zone the true value of effect estimate lies and therefore there is uncertainty over which decision to make (based on this outcome alone). The confidence interval is consistent with 2 decisions and so this is considered to be imprecise in the GRADE analysis and the evidence is downgraded by 1 level (‘serious imprecision’).

If the confidence interval of the effect estimate crosses into 3 zones, this is considered to be very imprecise evidence because the confidence interval is consistent with 3 clinical decisions and there is a considerable lack of confidence in the results. The evidence is therefore downgraded by 2 levels in the GRADE analysis (‘very serious imprecision’).

Implicitly, assessing whether the confidence interval is in, or partially in, a clinically important zone requires the Committee to estimate a minimally important difference (MID) or to say whether they would make different decisions for the 2 confidence limits.

Originally, the Committee was asked about MIDs in the literature or well-established MIDs in the clinical community (for example, international consensus documents) for the relevant outcomes of interest.

For the following review, the Committee agreed and used established MID:

Table 9

MIDs agreed by the Committee.

Due to the lack of well-established and widely accepted MIDs in the literature around cerebral palsy, the Committee agreed to use the GRADE default MIDs.

The Committee therefore considered it clinically acceptable to use the GRADE default MID to assess imprecision: a 25% relative risk reduction or relative risk increase was used, which corresponds to clinically important thresholds for a risk ratio of 0.75 and 1.25, respectively. This default MID was used for all the dichotomous outcomes in the interventions evidence reviews and for outcomes reported as ratios of means (RoM). For continuous outcomes, a MID was calculated by adding or subtracting 0.5 times standard deviations (SDS). For outcomes that were meta-analysed using the standardised mean difference approach (SMD), the MID was calculated by adding or subtracting 0.5 (given SD equals 1).

For the diagnostic questions, we assessed imprecision on the outcome of positive likelihood ratio because this was prioritised by the Committee as the most important diagnostic outcome for their decision-making. The assessment of imprecision for the results on positive likelihood ratio followed the same concept as used in interventional reviews. For example, if the 95% CI of the positive likelihood ratio crossed 2 zones (from moderately useful [5 to 10] to very useful [more than 10]) then imprecision was downgraded by 1, or if crossed 3 zones (not useful [less than 5], moderately useful [5 to 10] and very useful [more than 10]) then imprecision was downgraded by 2. These values have been used in previous guidelines developed in the NGA and the Committee agreed to using them. The specific use of a diagnostic test and which measures were to be of most interest (e.g. for rule in/rule out) were discussed with the Committee and recommendations were made accordingly.

3.3.3.7. Quality assessment of qualitative studies

Quality of qualitative studies (at study level) was assessed following the NICE checklists. The main quality assessment domains were organised across the definition of population included, the appropriateness of methods used and the completeness of data analysis and the overall relevance of the study participants to the population of interest for the guideline.

Individual studies were assessed for methodological limitations using an adapted Critical Appraisal Skills Programme (CASP 2006) checklist for qualitative studies, where items in the original CASP checklist were adapted and fitted into 5 main quality appraisal areas according to the following criteria:

• aim (description of aims and appropriateness of the study design)
• sample (clear description, role of the researcher, data saturation, critical review of the researchers’ influence on the data collection)
• rigour of data selection (method of selection, independence of participants from the researchers, appropriateness of participants)
• data collection analysis (clear description, how are categories or themes derived, sufficiency of presented findings, saturation in terms of analysis, the role of the researcher in the analysis, validation)
• results and findings (clearly described, applicable and comprehensible, theory production).

An adapted GRADE approach was then used to assess the evidence by themes across different included studies. Similar to GRADE in effectiveness reviews, this includes 4 domains of assessment and an overall rating:

• limitations across studies for a particular finding or theme (using the criteria described above)
• coherence of findings (equivalent to heterogeneity but related to unexplained differences or incoherence of descriptions)
• applicability of evidence (equivalent to directness, i.e. how much the finding applies to our review protocol)
• saturation or sufficiency (this related particularly to interview data and refers to whether all possible themes have been extracted or explored).

3.3.6.1. Literature review

The search strategy for existing economic evaluations combined terms capturing the target condition (cerebral palsy) and, for searches undertaken in MEDLINE, EMBASE and CCTR, terms to capture economic evaluations. No restrictions on language or setting were applied to any of the searches, but letters were excluded. Conference abstracts were considered for inclusion from January 2014, as high-quality studies reported in abstract form before this date were expected to have been published in a peer-reviewed journal. Full details of the search strategies are presented in Appendix E.

The Health Economist assessed the titles and abstracts of papers identified through the searches for inclusion using pre-defined eligibility criteria defined in Table 10.

Table 10

Inclusion and exclusion criteria for the systematic reviews of economic evaluations.

Once the screening of titles and abstracts was complete, full versions of the selected papers were acquired for assessment. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) for this search on economic evaluations is presented in Appendix F.

3.3.6.2. Undertaking new health economic analysis

As well as reviewing the published economic literature, as described above, new economic analysis was undertaken by the Health Economist in selected areas. The following priority areas for de novo economic analysis were agreed by the Committee after formation of the review questions and consideration of the available health economic evidence:

• determining the effective management of difficulties with saliva control (drooling) in children and young people with cerebral palsy
• interventions to reduce the risk of reduced bone mineral density and low-impact fractures in children and young people with cerebral palsy.

The methods and results of de novo economic analyses are reported in Appendix G. When new economic analysis was not prioritised, the Committee made a qualitative judgement regarding cost effectiveness by considering expected differences in resource and cost use between options, alongside clinical effectiveness evidence identified from the clinical evidence review.

3.3.6.3. Cost-effectiveness criteria

NICE’s report Social value judgements: principles for the development of NICE guidance sets out the principles that committees should consider when judging whether an intervention offers good value for money. In general, an intervention was considered to be cost effective if either of the following criteria applied (given that the estimate was considered plausible):

• the intervention dominated other relevant strategies (that is, it was both less costly in terms of resource use and more clinically effective compared with all the other relevant alternative strategies), or;
• the intervention cost less than £20,000 per QALY gained compared with the next best strategy, or;
• the intervention provided clinically significant benefits at an acceptable additional cost when compared with the next best strategy.

The Committee’s considerations of cost effectiveness are discussed explicitly in the ‘Consideration of economic benefits and harms’ section of the relevant sections.