Related NICE Health Technology Appraisals

Botulinum toxin type A for the prevention of headaches in adults with chronic migraine. NICE technology appraisal 260 (2012).

Related NICE Interventional Procedures

Deep brain stimulation for intractable trigeminal autonomic cephalalgias. NICE interventional procedure 381 (2011).

Percutaneous closure of patent foramen ovale for recurrent migraine. NICE interventional procedure guidance 370 (2010).

Related NICE Clinical Guidelines

Patient experience in adult NHS service. NICE clinical guideline 138 (2012).

The epilepsies. NICE clinical guideline 137 (2012).

Hypertension. NICE clinical guideline 127 (2011).

Anxiety. NICE clinical guideline 113 (2011).

Depression in adults. NICE clinical guideline 90 (2009).

Glaucoma. NICE clinical guideline 85 (2009).

Medicines adherence. NICE clinical guideline 76 (2009).

Stroke. NICE clinical guideline 68 (2008).

Head injury. NICE clinical guideline 56 (2007).

Referral guidelines for suspected cancer. NICE clinical guideline 27 (2005).

NICE is developing the following guidance (details available from www.nice.org.uk)

Head injury. NICE clinical guideline 56 (2007). Currently being updated. Publication tbc.

Referral guidelines for suspected cancer. NICE clinical guideline 27 (2005). Currently being updated. Publication expected March 2014.

2.7.1. Clinical literature search

Systematic literature searches were undertaken to identify evidence within published literature in order to answer the review questions as per The Guidelines Manual [2009]¹⁸¹. Clinical databases were searched using relevant medical subject headings, free-text terms and study type filters where appropriate. Studies published in languages other than English were not reviewed. Where possible, searches were restricted to articles published in English language. All searches were conducted on MEDLINE and Embase. The Cochrane Library was searched for all intervention questions. Additional subject specific databases were used for some questions: Cinahl for diaries, treatment questions and patient information; PsycINFO for education and self-management programmes, psychological therapies, medication over use headaches and patient information; AMED for non-pharmacological treatment of headaches. All searches were updated on 13 March 2012. No papers after this date were considered.

Search strategies were checked by looking at reference lists of relevant key papers, checking search strategies in other systematic reviews and asking the GDG for known studies. The questions, the study types applied, the databases searched and the years covered can be found in Appendix D.

During the scoping stage, a search was conducted for guidelines and reports on the websites listed below and on organisations relevant to the topic. A full list of websites is included in Appendix D. Searching for grey literature or unpublished literature was not undertaken. All references sent by stakeholders were considered.

• Guidelines International Network database (www.g-i-n.net)
• National Guideline Clearing House (www.guideline.gov/)
• National Institute for Health and Clinical Excellence (NICE) (www.nice.org.uk)
• National Institutes of Health Consensus Development Program (consensus.nih.gov/)
• National Library for Health (www.library.nhs.uk/).

2.7.2. Health economic literature search

Systematic literature searches were also undertaken to identify health economic evidence within published literature relevant to the review questions. The evidence was identified by conducting a broad search relating to the guideline population in the NHS economic evaluation database (NHS EED), the Health Economic Evaluations Database (HEED) and health technology assessment (HTA) databases with no date restrictions. Additionally, the search was run on MEDLINE, with a specific economic filter, from 2008, to ensure recent publications that had not yet been indexed by these databases were identified. Studies published in languages other than English were not reviewed. Where possible, searches were restricted to articles published in English language.

The search strategies for health economics are included in Appendix D. All searches were updated on 18 January 2012. No papers published after this date were considered.

2.8.1. Literature review

The process for review of evidence of effectiveness is as follows:

The Research Fellows:

• Identified potentially relevant studies for each review question from the relevant search results by reviewing titles and abstracts – full papers were then obtained.
• Reviewed full papers against pre-specified inclusion/exclusion criteria to identify studies that addressed the review question in the appropriate population and reported on outcomes of interest (review protocols are included in Appendix C, excluded studies lists are in Appendix O. The excluded studies list only details studies excluded after the full papers were ordered. Many would have previously been excluded when the titles and abstracts were reviewed.
• Critically appraised relevant studies using the appropriate checklist as specified in The Guidelines Manual¹⁸¹.
• Extracted key information about the study’s methods and results into evidence tables (evidence tables are included in Appendix E.
• Generated summaries of the evidence by outcome (included in the relevant chapter write-ups) and produced evidence statements indicating the number of included studies, sample size (number randomised), direction of effect, uncertainty and GRADE quality rating:
  • Randomised studies: meta analysed, where appropriate and reported in GRADE profiles (for clinical studies) – see below for details
  • Observational studies: data presented as a range of values in adapted GRADE profiles
  • Diagnostic studies: data presented as a range of values in adapted GRADE profiles
  • Prognostic studies: data presented as a range of values in adapted GRADE profiles
  • Qualitative studies: the quality of reporting for each study was summarised for three criteria in the guideline text: population, methods and analysis.

2.8.2. Inclusion/exclusion

See the review protocols in Appendix C for full details.

Note these key points:

The age range for this guideline was 12 years and over. Studies that included people younger than 12 were included only if the mean age of the population was over 12 years.

Crossover trials were only included in the review questions for acute treatment, however they were only included if it was clear from the paper that all participants included in the analysis had treated one headache attack only with each treatment, or if the data for the first crossover period only was available, in which case the study could be analysed as a parallel trial.

Placebo controlled trials were not included for the review question on the acute treatment of migraine as the GDG agreed that people seeking medical help for a migraine attack would have already tried over the counter medications. Therefore drug trials only were included if there was a head-to-head comparison.

The GDG agreed that for the majority of intervention review questions a sample size cut-off of 50 participants (25 per arm) was appropriate due to there being sufficient evidence with sample sizes greater than 50 which would provide a better estimate of the effect size. For most prognostic and diagnostic review questions, lager sample size cut-offs were applied (Chapters 5, 24 and 25). There were some exceptions in which lower sample size cut-offs were applied, or not cut-off values, when the GDG were aware that sufficient evidence at larger sample sizes would be lacking. These were:

• Indications for consideration of additional investigation (Chapter 4) – Minimum n=any
• Headache diaries for the diagnosis and management of primary headaches and medication overuse headache (Chapter 6) – Minimum n=any
• Imaging for diagnosis in people with suspected primary headache (Chapter 8.2) – Minimum n=any
• Imaging as a management strategy for people with suspected primary headaches (Chapter 8.3) – Minimum n=20 per arm
• Patient information and support (Chapter 9) – Minimum n=any
• Acute pharmacological treatment of cluster headache (Chapter 12) – Minimum n=any
• Prophylactic pharmacological treatment of cluster headache (Chapter 16) – Minimum n=any
• Prophylactic non-pharmacological management of primary headaches with psychological therapies (Chapter 19) – Minimum n=25 total
• Prophylactic non-pharmacological management of primary headaches with education and self management (Chapter 22) – Minimum n=25 total.

2.8.3. Methods of combining clinical studies

Appraising the quality of evidence by outcomes

The evidence for outcomes from the included RCT and observational studies were evaluated and presented using an adaptation of the ‘Grading of Recommendations Assessment, Development and Evaluation (GRADE) toolbox’ developed by the international GRADE working group (http://www.gradeworkinggroup.org/). The software (GRADEpro) developed by the GRADE working group was used to assess the quality of each outcome, taking into account individual study quality and the meta-analysis results. The summary of findings was presented as two separate tables in this guideline. The ‘Clinical/Economic Study Characteristics’ table includes details of the quality assessment while the ‘Clinical /Economic Summary of Findings’ table includes pooled outcome data, where appropriate, an absolute measure of intervention effect and the summary of quality of evidence for that outcome. In this table, the columns for intervention and control indicate the sum of the sample size for continuous outcomes. For binary outcomes such as number of people with an adverse event, the event rates (n/N: number of people with events divided by sum of number of people) are shown with percentages. Reporting or publication bias was only taken into consideration in the quality assessment and included in the Clinical Study Characteristics table if it was apparent. Each outcome was examined separately for the quality elements listed and defined in Table 2 and each graded using the quality levels listed in Table 3. The main criteria considered in the rating of these elements are discussed below (see section 2.8.4 Grading of Evidence). Footnotes were used to describe reasons for grading a quality element as having serious or very serious problems. The ratings for each component were summed to obtain an overall assessment for each outcome.

Table 2

Description of quality elements in GRADE for intervention studies.

Table 3

Levels of quality elements in GRADE.

2.8.4. Grading the quality of clinical evidence

After results were pooled, the overall quality of evidence for each outcome was considered. The following procedure was adopted when using GRADE:

1. A quality rating was assigned, based on the study design. RCTs start HIGH and observational studies as LOW, uncontrolled case series as LOW or VERY LOW.
2. The rating was then downgraded for the specified criteria: Study limitations, inconsistency, indirectness, imprecision and reporting bias. These criteria are detailed below. Observational studies were upgraded if there was: a large magnitude of effect, dose-response gradient, and if all plausible confounding would reduce a demonstrated effect or suggest a spurious effect when results showed no effect. Each quality element considered to have ‘serious’ or ‘very serious’ risk of bias were rated down −1 or −2 points respectively.
3. The downgraded/upgraded marks were then summed and the overall quality rating was revised. For example, all RCTs started as HIGH and the overall quality became MODERATE, LOW or VERY LOW if 1, 2 or 3 points were deducted respectively.
4. The reasons or criteria used for downgrading were specified in the footnotes.

The details of criteria used for each of the main quality element are discussed further in the following sections 2.8.5 to 2.8.8.

2.8.5. Study limitations

The main limitations for randomised controlled trials are listed in Table 4.

Table 4

Study limitations of randomised controlled trials.

The GDG agreed that wherever possible, except for acute pharmacological treatment of migraine (see chapter 11 for more information), comparators for intervention studies should be a placebo (or an active control for the case of non-pharmacological treatments) or another active intervention in a double blind situation. The GDG accepted that there were some non-pharmacological intervention studies were participant blinding was impossible or very hard to achieve in most situations (exercise, chapter 21, manual therapy, chapter 18, and education and self-management, chapter 22). Nevertheless, open-label studies for these intervention studies were downgraded to maintain a consistent approach in quality rating across the guideline; however, with interventions where a placebo or active control was possible, open label studies would be excluded.

Table 4 lists the limitations considered for randomised controlled trials.

2.8.6. Inconsistency

Inconsistency refers to an unexplained heterogeneity of results. When estimates of the treatment effect across studies differ widely (i.e. heterogeneity or variability in results), this suggests true differences in underlying treatment effect. When heterogeneity existed (Chi square p<0.1 or I-squared inconsistency statistic of >50%), but no plausible explanation can be found, the quality of evidence was downgraded by one or two levels, depending on the extent of uncertainty to the results contributed by the inconsistency in the results. In addition to the I- square and Chi square values, the decision for downgrading was also dependent on factors such as whether the intervention is associated with benefit in all other outcomes or whether the uncertainty about the magnitude of benefit (or harm) of the outcome showing heterogeneity would influence the overall judgment about net benefit or harm (across all outcomes).

If inconsistency could be explained based on pre-specified subgroup analysis, the GDG took this into account and considered whether to make separate recommendations based on the identified explanatory factors, i.e. population and intervention. Where subgroup analysis gives a plausible explanation of heterogeneity, the quality of evidence would not be downgraded.

2.8.7. Indirectness

Directness refers to the extent to which the populations, intervention, comparisons and outcome measures are similar to those defined in the inclusion criteria for the reviews. Indirectness is important when these differences are expected to contribute to a difference in effect size, or may affect the balance of harms and benefits considered for an intervention.

In this guideline the age range was people aged 12 and older. In cases where the population in the studies included children younger than 12, the studies were included if the average age was over 12, but the evidence would be down-graded for indirectness.

If the headache population included people with mixed headache types in the intervention reviews, the evidence would also be down-graded.

2.8.8. Imprecision

Imprecision refers to the certainty in the effect for the outcome. When results are imprecise or very imprecise we are uncertain if there is an important difference between interventions or not.

Assessing clinical importance

For the purposes of this guideline, clinical importance was assessed by comparing the effect estimate against the MID and reviewing the absolute effect reported in the GRADE summary table. For example, if the effect size was small (less than the MID), this finding suggests that there may not be enough difference to recommend one intervention over the other based on that outcome, unless in exceptional circumstances, the GDG agreed that the absolute effect was great enough to reach clinical importance. An effect estimate larger than the MID is considered to be clinically important.

Figure 1 illustrates how the clinical importance of effect estimates were considered along with imprecision. This is documented in the evidence statements throughout this guideline.

Figure 1

Illustration of precise and imprecise outcomes based on the confidence interval of outcomes in a forest plot in relation to the MID. Source: NCGC methods manual

Evidence statements

Evidence statements were formed for each outcome indicating the quantity and quality of evidence available, and the outcome and population to which they relate. Below are some examples to illustrate how the wording indicates the imprecision (uncertainty) and clinical importance:

• Precise, both the point estimate and confidence intervals are outside the MID :
  Xx studies with xx people showed that intervention a is more clinically effective than intervention b. [GRADE quality].
• Precise, both the point estimate and confidence intervals are between the MID and no difference:
  Xx studies with xx people showed that intervention a is more effective than intervention b, but the effect size was too small to be clinically important. [GRADE quality].
• Serious imprecision, point estimate outside the MID, and the confidence interval crosses the MID:
  Xx studies with xx people suggested that intervention a may be more clinically effective than intervention b, but there is some uncertainty. [GRADE quality].
• Serious imprecision, point estimate between the MID and no difference, and the confidence interval crosses the MID:
  Xx studies with xx people suggested that intervention a may be more effective than intervention b, but the effect size is too small to be clinically important, and there is some uncertainty. [GRADE quality].
• Very serious imprecision, point estimate outside the MID, and the confidence interval crosses the MID in both directions:
  Xx studies with xx people suggested that intervention a may be more clinically effective than intervention b, but there is considerable uncertainty. [GRADE quality].
• Very serious imprecision, point estimate between the MID and no difference, and the confidence interval crosses the MID in both directions:
  Xx studies with xx people suggested that intervention a may be more effective than intervention b, but the effect size is too small to be clinically important, and there is considerable uncertainty. [GRADE quality].
• Precise, point estimate close to line of no difference, confidence intervals just cross line of no difference:
  Xx studies with xx people showed that there is no difference between intervention a and intervention b. [GRADE quality].

When imprecision could not be assessed, the following statement will be used: “the difference is uncertain as no comparative analysis could be carried out”.

For diagnostic reviews, the imprecision was based on the outcome deemed to be most important, for example in cases where it was most important not to have a high number of false negative test results, the imprecision assessment would be based on specificity. No MID was defined for any of the diagnostic outcomes. The GDG were asked to review the evidence and agree the level of imprecision based on the confidence intervals around the effect size and absolute effect estimate.

2.9.1. Literature review

The Health Economist:

• Identified potentially relevant studies for each review question from the economic search results by reviewing titles and abstracts – full papers were then obtained
• Reviewed full papers against pre-specified inclusion / exclusion criteria to identify relevant studies (see below for details)
• Critically appraised relevant studies using the economic evaluations checklist as specified in The Guidelines Manual¹⁸¹
• Extracted key information about the study’s methods and results into evidence tables (evidence tables are included in Appendix E).
• Generated summaries of the evidence in NICE economic evidence profiles (included in the relevant chapter write-ups) – see below for details.

2.9.1.2. NICE economic evidence profiles

The NICE economic evidence profile has been used to summarise cost and cost-effectiveness estimates. The economic evidence profile shows, for each economic study, an assessment of applicability and methodological quality, with footnotes indicating the reasons for the assessment.

These assessments were made by the health economist using the economic evaluation checklist from The Guidelines Manual¹⁸¹. It also shows incremental costs, incremental outcomes (for example, QALYs) and the incremental cost-effectiveness ratio from the primary analysis, as well as information about the assessment of uncertainty in the analysis. See Table 5 for more details.

Table 5

Content of NICE economic profile.

Where economic studies compare multiple strategies, results are presented in the economic evidence profiles for the pair-wise comparison specified in the review question, irrespective of whether or not that comparison was ‘appropriate’ within the analysis being reviewed. A comparison is ‘appropriate’ where an intervention is compared with the next most expensive non-dominated option – a clinical strategy is said to ‘dominate’ the alternatives when it is both more effective and less costly. Footnotes indicate if a comparison was ‘inappropriate’ in the analysis.

For particular studies or original models comparing multiple strategies, results are not reported in the standard economic profile but are instead presented at the end of the relevant chapter in a paragraph summarising the study/model as a whole.

2.9.2. Undertaking new health economic analysis

As well as reviewing the published economic literature for each review question, as described above, new economic analysis was undertaken by the Health Economist in priority areas. Priority areas for new health economic analysis were agreed by the GDG after formation of the review questions and consideration of the available health economic evidence.

Additional data for the analysis was identified as required through additional literature searches undertaken by the Health Economist, and discussion with the GDG. Model structure, inputs and assumptions were explained to and agreed by the GDG members during meetings, and they commented on subsequent revisions.

See Appendices J and L for details of the health economic analyses undertaken for the guideline.

2.9.3. Cost-effectiveness criteria

NICE’s report ‘Social value judgements: principles for the development of NICE guidance’ sets out the principles that GDG members should consider when judging whether an intervention offers good value for money¹⁸⁰.

In general, an intervention was considered to be cost effective if either of the following criteria applied (given that the estimate was considered plausible):

1. The intervention dominated other relevant strategies (that is, it was both less costly in terms of resource use and more clinically effective compared with all the other relevant alternative strategies), or
2. The intervention cost less than £20,000 per quality-adjusted life-year (QALY) gained compared with the next best strategy.

If the GDG recommended an intervention that was estimated to cost more than £20,000 per QALY gained, or did not recommend one that was estimated to cost less than £20,000 per QALY gained, the reasons for this decision are discussed explicitly in the ‘from evidence to recommendations’ section of the relevant chapter with reference to issues regarding the plausibility of the estimate or to the factors set out in the ‘Social value judgements: principles for the development of NICE guidance’ ¹⁸⁰.

2.10.1. Research recommendations

When areas were identified for which good evidence was lacking, the guideline development group considered making recommendations for future research. Decisions about inclusion were based on factors such as:

• the importance to patients or the population
• national priorities
• potential impact on the NHS and future NICE guidance
• ethical and technical feasibility.

2.10.2. Validation process

The guidance is subject to a six week public consultation and feedback as part of the quality assurance and peer review the document. All comments received from registered stakeholders are responded to in turn and posted on the NICE website when the pre-publication check of the full guideline occurs.

2.10.3. Updating the guideline

Following publication, and in accordance with the NICE guidelines manual¹⁸¹, NICE will ask a National Collaborating Centre or the National Clinical Guideline Centre to advise NICE’s Guidance executive whether the evidence base has progressed significantly to alter the guideline recommendations and warrant an update.

2.10.4. Disclaimer

Health care providers need to use clinical judgement, knowledge and expertise when deciding whether it is appropriate to apply guidelines. The recommendations cited here are a guide and may not be appropriate for use in all situations. The decision to adopt any of the recommendations cited here must be made by the practitioners in light of individual patient circumstances, the wishes of the person, clinical expertise and resources.

The National Clinical Guideline Centre disclaims any responsibility for damages arising out of the use or non-use of these guidelines and the literature used in support of these guidelines.

2.10.5. Funding

The National Clinical Guideline Centre was commissioned by the National Institute for Health and Clinical Excellence to undertake the work on this guideline.