CRD summary

The authors concluded that there is little evidence about the reliability, validity and usefulness of track and trigger warning systems for identifying at-risk patients on the ward. Overall, the review appears to have been well-conducted and the authors’ conclusions about the limitations of the identified data seem appropriate.

Authors' objectives

To evaluate the reliability, validity and utility of physiological track and trigger warning systems (TTs) for identifying at-risk patients on the ward. The review also included studies that described TTs but these are not included in this abstract.

Searching

MEDLINE, MEDLINE In-Process and Other Non-indexed Citations, EMBASE, CINAHL, PsycINFO, the Cochrane Library and Web of Science were searched from 1990 to 2004 for studies published in full in English. Details of the search strategy were presented in a supplement. In addition, three relevant specified journals and the reference lists of key reports and reviews were screened, and experts and professional bodies were contacted for details of further studies. Datasets of TTs were obtained by contacting all acute National Health Service hospitals in England that have critical care facilities. In addition, study members, their contacts and authors of published studies were contacted for further data.

Study selection

Assessment of study quality

Studies were assessed using the criteria described by Laupacis et al., while the validity of clinical decision rules was assessed using criteria defined by McGinn et al. (see Other Publications of Related Interest, nos. 1-2). Datasets were assessed for methodological quality using the Quality of Assessment of Diagnostic Accuracy Studies (QUADAS) tool.

Data extraction

One reviewer extracted the data, which a second reviewer then checked. Logic, range and consistency checks were applied to check the validity of the datasets. Illogical values, outliers, cases where the anonymous unique patient identifier and date of admission were missing, and cases where a composite outcome could not be identified were removed.

Methods of synthesis

Results of the review

Five studies that reported the diagnostic accuracy of 4 TT systems (n=7,873) and 15 datasets (n=20,197) were included.

Diagnostic accuracy of the TTs reported in the studies.

None of the studies met all methodological criteria and none reported a measure of variability around estimates of diagnostic accuracy. None of the TTs met the requirements of a level 1 clinical decision rule (a rule validated in various settings with ‘confidence that it can change clinical behaviour and improve patient outcomes’). Where reported, estimates of sensitivity varied from 8 to 100% and estimates of specificity varied from 18 to 100%.

Diagnostic accuracy of the TTs from datasets.

Data were only received from 31 of the 92 hospitals that indicated they collected data and only 15 of these fulfilled the inclusion criteria. The diagnostic accuracy varied widely. Sensitivities and PPVs were low: the median sensitivity was 43.3 (quartiles: 25.4, 69.2) and the median PPV was 36.7 (quartiles: 29.3, 43.8). Specificities and NPVs were ‘generally acceptable’: the median specificity was 89.5 (quartiles: 64.2, 95.7) and the median NPV was 94.3 (quartiles: 89.5, 97.0). The area under the curve ranged from 0.61 to 0.84. Diagnostic accuracy was heterogeneous (p<0.001) among the 11 datasets with critical care follow-up. Meta-regression failed to identify the causes of this heterogeneity.

Authors' conclusions

There was little evidence about the reliability, validity and usefulness of TTs.

CRD commentary

The review question was defined in terms of the participants, intervention and outcomes. Primary studies and datasets were eligible. Several relevant sources were searched but no attempts were made to minimise publication or language bias. Methods were used to minimise reviewer error and bias in the study selection and data extraction processes, but it is unclear whether similar steps were taken for the assessment of validity. Validity was assessed and the methodological limitations of the primary studies discussed; although the validity of the datasets was apparently assessed, the results were not reported. In view of the diversity of the studies, it seems appropriate to have combined the primary studies in a narrative and combined data from datasets using medians, and explored sources of heterogeneity. Overall, the review appears to have been well-conducted and the authors’ conclusions about the limitations of the identified data seem appropriate.

Implications of the review for practice and research

Practice: The authors stated that, in view of the limited evidence about reliability, validity and utility, TTs should be used in conjunction with clinical judgement. Hospitals thinking about introducing TTs should take account of the latest evidence about the reliability and validity of systems, and should consider seeking a system that suits local conditions, is easy to use in practice and is acceptable to patients and staff.

Research: The authors stated that there is a need to validate TTs in their current settings. Large prospective studies are required to develop TTs that meet level I clinical decision rules (i.e. a rule validated in various settings with ‘confidence that it can change clinical behaviour and improve patient outcomes’).

Funding

UK National Health Service Research and Development Service Delivery & Organisation programme (SD0/74/2004).

Bibliographic details

Gao H, McDonnell A, Harrison D A, Moore T, Adam S, Daly K, Esmonde L, Goldhill D R, Parry G J, Rashidian A, Subbe C P, Harvey S. Systematic review and evaluation of physiological track and trigger warning systems for identifying at-risk patients on the ward. Intensive Care Medicine 2007; 33(4): 667-679. [PubMed: 17318499]

Other publications of related interest

1. Laupacis A, Sekar N, Stiell IG. Clinical prediction rules; a review and suggested modifications of methodological standards. JAMA 1997;277:488-94.

2. McGinn TG, Guyatt GH, Wyer PC, Naylor CD, Stiell IG, Richardson WS. Users' guides to the medical literature: XXII: how to use articles about clinical decision rules. JAMA 2000;284:79-84.

Indexing Status

Subject indexing assigned by NLM

MeSH

APACHE; Critical Care /methods; Databases, Factual; Emergency Medical Services /statistics & numerical data; Hospital Mortality; Humans; Intensive Care Units

AccessionNumber

12007005752

Database entry date

01/12/2008

Record Status

This is a critical abstract of a systematic review that meets the criteria for inclusion on DARE. Each critical abstract contains a brief summary of the review methods, results and conclusions followed by a detailed critical assessment on the reliability of the review and the conclusions drawn.