Table 131, Philips checklist for cross-infection - Cystic Fibrosis

Section	Pass/fail	Comments
Structure
Statement of decision problem / objective	P	There is a clear statement of the decision problem under consideration in the introduction section; specifically that the model will consider the economic impact vs no strategy of various strategies for the control of cross-infection in cystic fibrosis patients. Further detail on the population, interventions and pathogens is also included in the methods section. The objective of the analysis is consistent with this statement of the decision problem
Justification of modelling approach	P	There is a clear justification of the modelling strategy in the ‘methods: model structure’ section. No justification was given for the modelling framework selected, but this is consistent with Philips (2004) since “a model is simply and analytical framework with the purpose of synthesising the relevant evidence”
Statement of scope / perspective	P	The scope of the model is strictly defined by the NICE methods manual, although the author does highlight some key areas of uncertainty in the ‘methods: clinical effectiveness’ section. The model scope was heavily restricted by data availability, and this is reflected in the write-up
Structural assumptions	P	The structure of the model is consistent with a coherent theory of the natural history of the disease. The model structure does not describe a series of causal relationships between interventions and outcomes because of the confounding factor of non-clinic related infections, which is addressed in the model and therefore not relevant to the structural assumption check. The sources of data used to develop the model are clearly described and referenced and the model is independent of any particular model of service provision (although generic features of service provision are present in the model).
Strategies / Comparators	P	There is a clear definition of the data sources underpinning assumptions about the effectiveness of various strategies; the writeup notes that the data sources themselves are unclear on the exact procedure for enacting each different strategy. The strategies are not included in a statement of the decision problem, as there are multiple comparators. No detailed discussion of exclusions is recorded, but it is clearly implied that the evidence search was exhaustive and therefore feasible options which were not included in the model were not included for data availability reasons.
Model type	P	Deterministic decision tree is an entirely appropriate model for this decision question
Time horizon	P	The use of a non-standard one-year time horizon is clearly outlined in the text and justified strongly
Health states/disease pathways	P	The model uses paths in a decision tree model as the modelling substrate for disease states; the number and type of health states are clearly justified and recorded in the text
Cycle Length	N/A	Cycle length not relevant to a decision tree
Parsimony	P	In the view of the reviewer, the model is highly parsimonious, with the core decision tree being handled in a transparent way and the various costing ‘options boxes’ - although somewhat more complicated - clearly labelled and allowing important customisation options.
Data
Data Identification	P	Data identification is performed by a specialist information scientist
Data Synthesis	P	Data has been synthesised using standard methods for a deterministic decision tree. The only notable departure from standard methodology is the synthesis of incidence and prevalence results, which is justified in the write-up and made necessary by the data sources reporting different outcome measures.
Discounting	P	No discount rate was applied. This decision is justified in the write-up and consistent with the NICE Methods Manual
Analysis of trial data	P	It was not possible to analyse the trials included at the patient level. ITT-type considerations are not relevant to this model, although a discussion of similar issues occurs around the Thornton 2005 paper, where patients randomised to the ‘hospital’ group did not receive all of their care in hospital
Treatment effects	P	This is not strictly relevant as trials reported the absolute probability of infection (or absolute prevalence of infection, sometimes). Nevertheless the handling of these data are appropriate in the model.
Transition probabilities	P	Transition probabilities are simple to calculate in a decision tree, and are handled appropriately in this model. The probabilities are given in the write-up
Mortality	N/A	The time horizon of this model means that all-cause mortality is not relevant to the decision problem. This is not explained in the write-up.
Extrapolation	P	Several assumptions of this sort exist in the model (for example that patients with a terminal chronic infection will die at exactly halfway through their treatment costs). Most of these assumptions are justified with reference to the Guideline committee. This is appropriate, because the modelling supports their work and draws on their expert opinion.
Risk factors	N/A	No risk factors were included in the model as the data could not support such additions. The results indicate that such risk factors are probably not relevant to the decision problem.
Utilities	P	HRQoL, the sources of information on HRQoL and analysis of the different possible modelling choices are carefully described in the section ‘1.6 Methods: health-related quality of life’
Charges and costs	P	Resource use is described in section ‘1.5 Methods: resource and cost use’. The model uses mostly entirely standard sources (PSSRU or NHS Reference Costs), with some nonstandard sources such as academic literature and NHS Estates data. The most uncertain resource tariff (single vs shared-occupancy rooms) was well grounded in the literature and the justification for using weak evidence was robust throughout.
Adjustment over time / between countries	P	No adjustment has been made between countries as this was not relevant. Adjustment between time periods has been made based on the hospital & community health services (HCHS) index, which is a standard method
Half-cycle correction	N/A	Half cycle correction would not be standard methodology for a decision tree
Data incorporation	P	The model appears to be internally consistent with respect to its choice of measurement units, time intervals and population characteristics. The sources of data are clearly described in the ‘methods’ sections (most explicitly in section ‘1.4 Methods: clinical effectiveness’), with sufficient discussion to allow for an intelligent assessment of the data quality.
Uncertainty
General statement regarding sensitivity analysis	P	The write-up includes a general statement outlining the strategy for sensitivity analysis
Structural	N/A	The general form of a deterministic decision tree is clearly the most appropriate for performing this kind of analysis, and so it would not improve the model to attempt re-analysis using a different structure
Methodological	N/A	Methodological uncertainty cannot systematically be explored in a NICE cost-effectiveness analysis; the analyst is constrained by the Reference Case
Parameter	P	The model is deterministic, and parameter uncertainty over estimates of effectiveness do not appear to have been translated into the model. Nevertheless, key values have been varied in sensitivity analysis, limiting the extent to which the model could be criticised for not incorporating parameter uncertainty. This sensitivity is largely univariate and / or scenario modelling, which explores a plausible range of parameters for particularly uncertain or important values. It is not clear from the write-up if the author has used results from the sensitivity analysis to identify areas where the value of future information is high, although this is not the principle point of a NICE guideline.
Consistency
Internal	P	The model behaves as theoretical predictions predict it should - values which should increase cost-effectiveness appear to do so and values which should do the opposite appear to do that. There do not appear to be any values with zero effect on the outcome, suggesting the model logic is piping through correctly. Extreme values - including zero values - do not produce contradictory or ridiculous results.
External	P	The model is extremely amenable to straightforward explanation and its structure is very clear. The output of the model appears to largely track committee opinion as to the relative costs of various interventions, although there is no health economic literature addressing the question this model answers so no independent way of corroborating this
Between-model	P	There is no health economics literature addressing this issue, so no independent between-model corroboration. A simple replication attempt produces consistent results with the final model, suggesting a high degree of between-model reliability
Predictive	N/A	There is no way to test whether the model has predictive validity before publication

Section

Pass/fail

Comments

Structure

Statement of decision problem / objective

There is a clear statement of the decision problem under consideration in the introduction section; specifically that the model will consider the economic impact vs no strategy of various strategies for the control of cross-infection in cystic fibrosis patients. Further detail on the population, interventions and pathogens is also included in the methods section. The objective of the analysis is consistent with this statement of the decision problem

Justification of modelling approach

There is a clear justification of the modelling strategy in the ‘methods: model structure’ section. No justification was given for the modelling framework selected, but this is consistent with Philips (2004) since “a model is simply and analytical framework with the purpose of synthesising the relevant evidence”

Statement of scope / perspective

The scope of the model is strictly defined by the NICE methods manual, although the author does highlight some key areas of uncertainty in the ‘methods: clinical effectiveness’ section. The model scope was heavily restricted by data availability, and this is reflected in the write-up

Structural assumptions

The structure of the model is consistent with a coherent theory of the natural history of the disease. The model structure does not describe a series of causal relationships between interventions and outcomes because of the confounding factor of non-clinic related infections, which is addressed in the model and therefore not relevant to the structural assumption check. The sources of data used to develop the model are clearly described and referenced and the model is independent of any particular model of service provision (although generic features of service provision are present in the model).

Strategies / Comparators

There is a clear definition of the data sources underpinning assumptions about the effectiveness of various strategies; the writeup notes that the data sources themselves are unclear on the exact procedure for enacting each different strategy. The strategies are not included in a statement of the decision problem, as there are multiple comparators. No detailed discussion of exclusions is recorded, but it is clearly implied that the evidence search was exhaustive and therefore feasible options which were not included in the model were not included for data availability reasons.

Model type

Deterministic decision tree is an entirely appropriate model for this decision question

Time horizon

The use of a non-standard one-year time horizon is clearly outlined in the text and justified strongly

Health states/disease pathways

The model uses paths in a decision tree model as the modelling substrate for disease states; the number and type of health states are clearly justified and recorded in the text

Cycle Length

N/A

Cycle length not relevant to a decision tree

Parsimony

In the view of the reviewer, the model is highly parsimonious, with the core decision tree being handled in a transparent way and the various costing ‘options boxes’ - although somewhat more complicated - clearly labelled and allowing important customisation options.

Data

Data Identification

Data identification is performed by a specialist information scientist

Data Synthesis

Data has been synthesised using standard methods for a deterministic decision tree. The only notable departure from standard methodology is the synthesis of incidence and prevalence results, which is justified in the write-up and made necessary by the data sources reporting different outcome measures.

Discounting

No discount rate was applied. This decision is justified in the write-up and consistent with the NICE Methods Manual

Analysis of trial data

It was not possible to analyse the trials included at the patient level. ITT-type considerations are not relevant to this model, although a discussion of similar issues occurs around the Thornton 2005 paper, where patients randomised to the ‘hospital’ group did not receive all of their care in hospital

Treatment effects

This is not strictly relevant as trials reported the absolute probability of infection (or absolute prevalence of infection, sometimes).
Nevertheless the handling of these data are appropriate in the model.

Transition probabilities

Transition probabilities are simple to calculate in a decision tree, and are handled appropriately in this model. The probabilities are given in the write-up

Mortality

N/A

The time horizon of this model means that all-cause mortality is not relevant to the decision problem. This is not explained in the write-up.

Extrapolation

Several assumptions of this sort exist in the model (for example that patients with a terminal chronic infection will die at exactly halfway through their treatment costs). Most of these assumptions are justified with reference to the Guideline committee. This is appropriate, because the modelling supports their work and draws on their expert opinion.

Risk factors

N/A

No risk factors were included in the model as the data could not support such additions. The results indicate that such risk factors are probably not relevant to the decision problem.

Utilities

HRQoL, the sources of information on HRQoL and analysis of the different possible modelling choices are carefully described in the section ‘1.6 Methods: health-related quality of life’

Charges and costs

Resource use is described in section ‘1.5 Methods: resource and cost use’. The model uses mostly entirely standard sources (PSSRU or NHS Reference Costs), with some nonstandard sources such as academic literature and NHS Estates data. The most uncertain resource tariff (single vs shared-occupancy rooms) was well grounded in the literature and the justification for using weak evidence was robust throughout.

Adjustment over time / between countries

No adjustment has been made between countries as this was not relevant. Adjustment between time periods has been made based on the hospital & community health services (HCHS) index, which is a standard method

Half-cycle correction

N/A

Half cycle correction would not be standard methodology for a decision tree

Data incorporation

The model appears to be internally consistent with respect to its choice of measurement units, time intervals and population characteristics. The sources of data are clearly described in the ‘methods’ sections (most explicitly in section ‘1.4 Methods: clinical effectiveness’), with sufficient discussion to allow for an intelligent assessment of the data quality.

Uncertainty

General statement regarding sensitivity analysis

The write-up includes a general statement outlining the strategy for sensitivity analysis

Structural

N/A

The general form of a deterministic decision tree is clearly the most appropriate for performing this kind of analysis, and so it would not improve the model to attempt re-analysis using a different structure

Methodological

N/A

Methodological uncertainty cannot systematically be explored in a NICE cost-effectiveness analysis; the analyst is constrained by the Reference Case

Parameter

The model is deterministic, and parameter uncertainty over estimates of effectiveness do not appear to have been translated into the model. Nevertheless, key values have been varied in sensitivity analysis, limiting the extent to which the model could be criticised for not incorporating parameter uncertainty. This sensitivity is largely univariate and / or scenario modelling, which explores a plausible range of parameters for particularly uncertain or important values. It is not clear from the write-up if the author has used results from the sensitivity analysis to identify areas where the value of future information is high, although this is not the principle point of a NICE guideline.

Consistency

Internal

The model behaves as theoretical predictions predict it should - values which should increase cost-effectiveness appear to do so and values which should do the opposite appear to do that. There do not appear to be any values with zero effect on the outcome, suggesting the model logic is piping through correctly. Extreme values - including zero values - do not produce contradictory or ridiculous results.

External

The model is extremely amenable to straightforward explanation and its structure is very clear. The output of the model appears to largely track committee opinion as to the relative costs of various interventions, although there is no health economic literature addressing the question this model answers so no independent way of corroborating this

Between-model

There is no health economics literature addressing this issue, so no independent between-model corroboration. A simple replication attempt produces consistent results with the final model, suggesting a high degree of between-model reliability

Predictive

N/A

There is no way to test whether the model has predictive validity before publication

From: Appendix K, Health Economics

Cystic Fibrosis: Diagnosis and management.

NICE Guideline, No. 78.

National Guideline Alliance (UK).

London: National Institute for Health and Care Excellence (NICE); 2017 Oct 25.

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Table 131Philips checklist for cross-infection