Study designs

The 24 studies included in the systematic review had the following designs. Most (20) were prospective before-and-after studies and these involved either single critical care units,^{50,51,87,93,94,97,108,110,138} multiple critical care units for which results were pooled across the units,^{34,83,98,126,129,130,135,139} or multiple critical care units for which results were presented separately for each unit.^{103,109,122,144} One RCT¹³⁶ was based on clusters, with hospitals randomised to interventions, but implementation was at the level of the critical care unit, comparing ‘Virtual Collaborative’ and ‘Toolkit’ CQI programmes in 60 critical care units. One non-randomised study⁶⁸ compared a CQI intervention and control (no intervention) in 17 critical care units. The remaining two studies^83,117 were single-cohort CQI programmes without a true baseline period in which data from early in the intervention period served as the baseline (Table 15).

TABLE 15

Summary of study characteristics

Populations

Characteristics of the critical care patient populations were generally reported superficially. Only 10 out of the 24 studies (42%)^{50,87,93,103,109,110,129,136,139,144} reported the age and/or sex of their patient population (Table 16). The youngest patients reported were on average in their early 40s, in the studies by DuBose and colleagues⁹³ and Higuera and colleagues.¹⁰³ The oldest critical care population group reported was in the study by Rosenthal and colleagues,¹²⁹ in which the mean age was close to 72 years. Where reported, the studies included mixed-sex populations, with the proportion of men ranging from 46% to 79%. Only one study mentioned the patients' ethnicity, stating that 94% of the population was Caucasian.¹³⁹ Comorbidities were reported inconsistently, with some studies providing considerable detail and others providing no information (see data extractions in Appendix 5). In most studies the population characteristics either did not differ between the intervention and comparator groups or were not reported. Where differences between groups were evident we considered the risk of bias (reported below).

TABLE 16

Patient characteristics in 10 studies that reported patients’ age and/or sex

The critical care specialties most frequently reported were medical, surgical and cardiac. Three of the regional-scale studies^33,83,136 were not conducted entirely in adult critical care units but we judged the studies to have met the population inclusion criterion for the systematic review because the proportion of non-adult critical units was small. Burrell and colleagues⁸³ included two paediatric critical care units (95% were adult units), Pronovost and colleagues³⁵ included one paediatric unit (99% were adult units) and Speroff and colleagues¹³⁶ included two paediatric units in each study arm (93% were adult units). Palomar Martinez and colleagues⁶⁸ appeared to focus on adult critical care but did not state this explicitly (the study authors were contacted but had not clarified this at the time of writing).

Vascular devices

All 24 studies included in the systematic review reported that they used CVCs, but it was not always clear whether other catheter types were also used. Misset and colleagues¹¹⁷ included arterial catheters, whereas Warren and colleagues⁵¹ specifically excluded these. Antimicrobial impregnated catheters were used routinely by DuBose and colleagues⁹³ and Warren and colleagues¹³⁹ but were not used by Sherertz and colleagues¹³⁵ or Warren and colleagues.⁵¹ Longmate and colleagues¹¹⁰ specified that their default catheters were non-impregnated and had four or five lumens, whereas Guerin and colleagues⁹⁸ used antimicrobial-impregnated catheters unless patients were hypersensitive to them. Coopersmith and colleagues^50,87 used four-lumen antimicrobial-impregnated catheters before their educational intervention but stated that their ‘accessibility was limited’ after the implementation of education (we discuss the implications of this for risk of bias below). The only other studies that reported lumen characteristics were by Wall and colleagues,¹³⁸ who used triple-lumen catheters; Render and colleagues,¹²⁶ who specified that 60% of the catheters were multilumen; and Misset and colleagues,¹¹⁷ who specified that single-lumen or multilumen catheters were used as clinically required.

Definitions of bloodstream infections

Of the 24 studies, 12 defined their infections as, or equivalent to, catheter-associated (CABSI), 11 defined their infections as, or equivalent to, catheter-related (CRBSI), and the remaining study¹¹⁷ gave an unclear definition (Table 17). Only eight studies provided definitions that agree with those of the Matching Michigan programme. In one of these studies, by Palomar Martinez and colleagues,⁶⁸ definitions equivalent to CABSI and CRBSI were both accepted but were not separated when reporting the results (see Table 17). In support of their reported infection definitions, 28 of the studies also cited published references to definitions of CABSI or CRBSI. However, the most frequently-cited of these references, by Garner and colleagues,¹⁴⁷ does not actually define CABSI or CRBSI.

TABLE 17

Comparison of catheter-associated and CRBSI definitions in the primary studies with definitions used in the Matching Michigan programme in England

Types of educational intervention

The types of educational approach used in the 24 included studies are summarised in Table 15. As noted in the protocol (see Appendix 1), educational interventions were defined in a broad sense to capture any type of information provision relating to the prevention of catheter-BSI by critical care staff, including the use of checklists, performance feedback and information surveillance feedback. Fourteen of the studies (58%)^{50,51,87,93,94,108,109,122,129,130,135,138,139,144} used interventions that we classified as purely educational. These ranged from the provision of single lectures^109,122 to multimodal combinations of bedside teaching,^51,94,139 in-services ^50,87,94,139 self-study,^50,51,139 practical demonstrations,^87,94,144 slide shows or videos,^94,144 lectures,^{51,87,109,139} discussion groups or classes,^51,130 supervision,¹³⁸ simulations,¹³⁵ and/or posters and fact sheets^{50,51,87,108,139} (see Table 15). In one study¹²⁹ the educational approach was not reported and it is possible that other studies may not have fully reported all of the educational approaches that they used. These purely educational interventions were all implemented at a local scale, mostly in single critical care units.

A total of 10^{34,68,83,97,98,103,110,117,126,136} of the 24 studies (42%) were classified as having intervention components beyond education (a classification previously used by Safdar and Abad³⁷ in a systematic review of interventions for preventing health care-associated infections). These studies included some of the educational approaches referred to above but, in addition, they used changes in equipment (e.g. the provision of a catheter supplies cart or alcohol for skin antisepsis) or infrastructure (e.g. the provision of a team). Five^{34,68,83,126,136} of these studies implemented their interventions at a regional scale, in 8,¹²⁶ 17,⁶⁸ 37,⁸³ 60¹³⁶ or 103³⁴ critical care units (see Table 15). These regional-scale studies included the Michigan ‘Keystone ICU’ project conducted in the USA;^34,123,124 the ‘CLAB ICU’ project,⁸³ which sought to replicate aspects of the Keystone ICU project in Australia; the ‘Bacteraemia Zero’ project,⁶⁸ which sought to replicate aspects of the Keystone ICU project in Spain, and the RCT referred to above, which compared Virtual Collaborative and Toolkit approaches in the USA.¹³⁶ Four of these interventions could be described as CQI programmes^{34,83,126,136} in which iterative improvements in clinical practices became embedded over time, whereas the fifth⁶⁸ was a short (3-month) pilot study of a CQI programme. A further two CQI programmes were implemented at a local scale in individual critical care units in France¹¹⁷ and Scotland.¹¹⁰ The remaining three studies^97,98,103 that implemented interventions with components beyond education at a local scale (in one to three critical care units) included a catheter insertion care bundle,⁹⁷ a catheter ongoing care bundle,⁹⁸ and an educational intervention with provision of alcohol hand rub.¹⁰³

The duration of interventions included in the systematic review ranged from a brief 15-minute lecture reported by Perez Parra and colleagues¹²² to up to 6 years in the case of a multimodal educational intervention reported by Eggimann and colleagues^94,95 (see Table 15). With the exception of five studies,^{51,108,122,135,139} the studies monitored effects of the interventions on catheter-BSI incidence density only during the period of intervention implementation, without post-intervention follow-up. This reflects an intention in many of the studies that the interventions would become embedded into routine clinical practice.

Formal and informal education

Formal education implies that participants set aside some time for structured learning, for example to participate in classes, lectures, seminars, view slide shows, or take self-study modules. Informal education may involve passive information dissemination, for example in-service discussions during daily rounds, posters, newsletters, or supervision. The distinction is important from the perspective of resource provision since staff engaging in formal learning will be taken away from their critical care duties. Nearly all of the educational interventions contained formal education components (Table 18), implying that staff cover would need to be provided for those staff participating in an intervention. However, the concentration of the education (duration and frequency of sessions and whether they were periodically reinforced) was rarely reported (see data extraction forms in Appendix 5).

TABLE 18

Formal education approaches used in the primary studies

Educational theory

None of the 24 included studies specified an educational or behavioural theory. Pronovost and colleagues¹²⁴ referred to the ‘4E’ framework proposed by Heifetz, in which the technical aspects (i.e. scientific evidence; definitions of measures standardised across hospitals) are classed as education and evaluation, whereas the adaptive elements (i.e. implementation of measures; interventions modified to fit the local context of a clinical area) are classed as engagement and execution. The authors stated only that the technical functions in the study were centralised.

Comparison of educational interventions with UK practice

The Matching Michigan programme is reflective of current NHS practice for preventing catheter-BSI (see Chapter 1) and educational interventions included in the systematic review shared some of the approaches used in Matching Michigan to varying degrees (Table 19). Of the 24 included studies, 11 studies^{34,68,83,93,97,108–110,126,136,138} used checklists to improve compliance with best practices for prevention of catheter-BSI. Three studies^34,83,98 empowered their staff to halt CVC insertion procedures if protocols for infection prevention were not followed. Fourteen studies^{34,50,83,87,97,108–110,117,126,130,136,138,139} included infection surveillance feedback in their interventions, in which catheter-BSI incidence rates were reported to critical care staff at regular intervals, either at meetings or using posters or fact sheets. However, for six of these studies, infection surveillance feedback was already in place during the baseline period and would not explain any differences in catheter-BSI incidence densities observed between baseline and intervention periods. Fourteen studies involved interventions that provided some form of performance feedback to the critical care unit staff. The feedback primarily provided information on compliance with interventions or specific intervention components (e.g. hand hygiene¹¹⁰), but in some studies feedback was given on problems encountered¹⁰⁹ or on staff competence assessed through tests of knowledge or skills, with staff required to repeat training if satisfactory scores were not achieved.^50,51,98 In three studies^68,122,139 the critical care staff were given tests or questionnaires, but it is unclear whether the results were fed back to the staff.

TABLE 19

Strategies used in educational interventions for prevention of catheter-BSI compared with the Matching Michigan programme

Clinical practices recommended in the Department of Health ‘High Impact Intervention No. 1’ for CVC insertion and maintenance²⁸ (see Tables 2 and 3) and used in Matching Michigan were included to varying degrees in the primary studies (Table 20). Where studies had two interventions,^109,136 the interventions within each study did not differ in the clinical practices addressed and have been summarised together in Table 20. The most frequently addressed of the recommended clinical practices were hand hygiene prior to patient contact, and the use of maximal sterile barrier precautions and antiseptic insertion site preparation. Only three studies^93,129,139 did not address hand hygiene practices, although in five studies^{68,97,122,130,135} there was ambiguity as to whether hand hygiene was prior to or after contact with the patient (see Table 20). Generally, fewer studies addressed practices related to CVC ongoing care than practices related to CVC insertion. When interpreting Table 20, it is important to bear in mind that authors of the primary studies might not have reported all of the clinical practices that their interventions addressed, as studies were variable in the amount of detail they provided about their methods.

TABLE 20

Evidence-based practices specified the UK ‘High Impact Intervention No. 1’ for CVC insertion and maintenance that were included in the primary studies

The most comprehensive interventions in terms of the number of evidence-based clinical practices they included were education-only local-scale interventions reported by Eggimann and colleagues,⁹⁴ Lobo and colleagues¹⁰⁸ and Warren and colleagues,⁵¹ and regional-scale interventions with components beyond education reported by Burrell and colleagues⁸³ and Pronovost and colleagues.³⁴ Among these studies, the Burrell study (the CLAB ICU project)⁸³ appears most relevant to current UK clinical practice, as it sought to replicate the Keystone ICU project at a regional scale.

Summary of study characteristics

Twenty-four studies were included in the systematic review. Most studies were conducted in medical, surgical and cardiac critical care units but the studies provided little information about their study populations and the vascular devices used. Where reported, patients' ages ranged from early 40s to early 70s. Half of the studies were conducted in the USA,^{34,50,51,87,93,97,98,126,135,136,138,139} with only one study conducted in the UK¹¹⁰ (a CQI programme in a single critical care unit in Scotland). The majority of studies used a single-cohort uncontrolled before-and-after design, with only one RCT included. The interventions evaluated in the studies were diverse in their educational approaches, ranging from single lectures in single critical care units to regional-scale CQI programmes in up to 103 critical care units. Fifty-eight per cent of the interventions reported were purely educational and 42% included components beyond education. Nearly all interventions included formal education approaches that would take staff away from bedside patient care. The interventions varied in the extent to which they addressed evidence-based clinical practices relevant to the NHS for preventing catheter-BSI. Most interventions focused on catheter insertion rather than catheter ongoing care, with hand hygiene, maximal barrier precautions and antiseptic preparation of the insertion site being the most frequently addressed practices. Different educational approaches, which were unique to individual studies, were used to address the same clinical practices. Although studies had to provide a definition of catheter-BSI to be included in the systematic review, only eight studies provided definitions consistent with those used in the NHS for the Matching Michigan programme. Two of the regional-scale interventions^68,83 aimed to replicate the Keystone ICU project intervention in different countries, and, of these, the CLAB ICU project⁸³ appears most relevant to NHS practice, although it was conducted in an Australian health-care setting.

Quality assessment: risk of bias

The included primary studies were difficult to assess for risk of bias because in most cases insufficient information was reported, resulting in a judgement of ‘unclear’ (Table 21). Methodological information supporting the judgements reached by the reviewers is given in the data extraction forms (see Appendix 5). When interpreting risk of bias it is important to bear in mind that, as judgements of bias are constrained by the availability of information, studies classified as having high or low risk of bias may not necessarily be at greater or lower risk of bias than those studies classified as unclear.

TABLE 21

Quality assessment: risk of bias – criteria adapted for before-and-after studies

Studies judged to be at high risk of bias

Three studies^68,109,144 were judged to be at high risk of selection bias, as the numbers of patient-days and CVC-days were consistently lower in the intervention period than in the pre-intervention period;¹⁰⁹ baseline incidence of catheter-BSI was higher in control than in intervention critical care units;⁶⁸ or the McCabe rapid fatality score, number of trauma patients and number of patients who had a CVC inserted were statistically significantly higher in the intervention than in the baseline period.¹⁴⁴

Four studies^{68,109,129,130} were judged to be at high risk of performance bias owing to a potential influence of other concurrent or overlapping interventions as the planned intervention overlapped with different hand hygiene interventions;^129,130 staff in a critical care unit receiving one intervention moved to a different critical care unit receiving another intervention;¹⁰⁹ or staff in control critical care units were able to attend training for the intervention.⁶⁸

Two studies^50,87 were judged to be at high risk of performance bias owing to staff or policy changes because they reported changes in antibiotic prescribing patterns and accessibility of quadruple-lumen catheters,⁵⁰ or reported that the number of ICU beds was higher in the intervention period (six beds; 33% increase).⁸⁷

Four studies^50,51,93,110 were judged to be at high risk of performance bias because critical care staff assessing outcomes (the staff collecting and culturing blood samples and analysing the results) were not blinded. The remaining 19 studies did not report whether the staff were blinded but it seems unlikely that authors would have blinded study participants without reporting so.

One study⁵⁰ was judged to be at high risk of performance bias, as the authors implied that there was a procedural difference in diagnosis and/or reporting of infections between the baseline and intervention periods (it was stated that treatment of catheter colonisation could have accounted for a large number of documented infections in the pre-intervention time period).

Studies judged to be at low risk of bias

Studies were judged to be at low risk of bias if their population characteristics did not appear to differ between baseline and intervention periods^94,103,139 (low selection bias risk); they stated that no other concurrent interventions were conducted,^94,97,122 or no changes in critical care structure or staffing occurred^94,117 that could potentially affect catheter-BSI incidence; they stated that catheter-BSI definitions and diagnosis procedures did not change^93,122 (low performance bias risk); or missing data were deducible and did not appear to differ between baseline and intervention periods^94,144 (low attrition bias risk).

Risk of bias in randomised controlled trials

Only one RCT, by Speroff and colleagues,¹³⁶ was included in the systematic review. In addition to the risk of bias criteria reported above for before-and-after studies, which we used to assess before-and-after comparisons within each of the RCT arms, we assessed the risk of selection bias, according to the adequacy of randomisation and allocation concealment, using the Cochrane Collaboration criteria for risk of bias in RCTs.⁵⁸ The RCT¹³⁶ was judged to be at unclear risk of selection bias because insufficient information was provided about the methods of random sequence generation and allocation concealment.

Other bias risk

Information collected in the data extraction forms (see Appendix 5) suggests that studies may have been at risk of self-reporting bias (study outcomes were reported directly by the investigators without independent checking). For example, self-reporting of outcomes was routine^110,136 or dependent upon staff availability;⁸³ an audit tool and intervention were implemented by the same team that designed them;⁸⁷ the study co-ordinator who was involved in delivering an intervention checked data sheets for missing items and errors;¹²⁹ and it was assumed that nurses accurately captured information and completed a checklist for every insertion, without formal validity and reliability analyses.¹³⁸

In three studies^110,117,136 the intervention as reported by the authors was not intended solely for the prevention of catheter-BSI: Longmate and colleagues¹¹⁰ reported an intervention for preventing CRBSI, VAP and MRSA; Misset and colleagues¹¹⁷ reported an intervention for preventing CRBSI, VAP and UTIs; and Speroff and colleagues¹³⁶ reported two interventions, each for preventing both central line-associated bloodstream infection (CLABSI) and VAP. In these studies, clinical effectiveness must be evaluated for the intervention as a whole, as effects of the individual intervention components that target the different types of infections are not separable.

Reporting of data collection in the primary studies

Of the 24 primary studies,^{34,50,51,68,83,87,93–95,97,103,108–110,117,122–124,126,129,130,135,136,138,139,144,146} none provided full details of their data collection processes. The most detailed descriptions of data collection were reported by Burrell and colleagues⁸³ and Wall and colleagues,¹³⁸ but even their descriptions were incomplete. Burrell and colleagues^83,146 stated that data entry was manual, with an infection nurse checking data on each form. Collected data were received and collated by the Clinical Excellence Commission. Missing and invalid data were followed up and validity of reported central line-associated bacteraemia (CLAB) were confirmed with individual critical care units. However, many critical care units did not have microbiological support and reported CLAB through discussion with senior critical care staff, while improved understanding of surveillance definitions versus clinical definitions led to some CLAB cases being reclassified. Wall and colleagues¹³⁸ reported that upon completing a checklist, a nurse detached the top page (with all items readable) and dropped it in a secure lockbox. The second page remained on the patient's chart with the sensitive items blacked out. The infection control practitioners collected the checklists daily and scanned the de-identified forms into a pre-established computerised database using scanning software which read pre-established fields on the checklist into a spreadsheet database. These data were stored on a secure computer at the Center for Clinical Improvement for future statistical analyses.

Only one⁹⁴ of the 24 studies reported that their data collection approach was reliable (pre-tested and standardised in several pilot phases), for infection surveillance. One study⁹⁸ mentioned that device-day data collected by critical care unit nursing staff were compared with data collected daily by the intravascular catheter management team to confirm the accuracy of data collection but did not provide any supporting data. Three studies^110,136,137 stated explicitly that validity and/or reliability of the data collection method was not assessed. It seems improbable that validity and reliability of data collection would have been assessed in the remaining studies, as no mention was made in the publications.

Summary of quality assessment

Overall, the methodological quality of the 24 included primary studies was difficult to assess owing to limited or unclear reporting of study populations and research methods. Nine studies^{50,51,68,87,93,109,110,129,130} were judged to be at high risk of bias, but, as risk of bias could be assessed only for well-reported studies, the extent of risks of bias among the studies may have been underestimated. Several different types of data were collected in the primary studies, including infection surveillance information, results of tests and assessments, and information on patient outcomes. However, none of the included studies fully reported its data collection methods and the majority of studies did not report whether data collection approaches had been shown to be valid and/or reliable. In most cases the staff who were involved in data collection were not specified, and studies may have been at risk of self-reporting bias (we did not formally assess whether staff involved in data collection were independent, but the authors of five studies^{83,87,110,129,136} implied that they were not).

Incidence density of catheter-bloodstream infection

In 10 studies, insufficient data were reported for the calculation of incidence density RRs with 95% CIs or there were ambiguities in the published data,^{83,93,94,97,117,126,130,135,136,138} and we attempted to contact the authors for clarification. Six authors responded with information, three did not respond and one responded stating that relevant data were not available. Data that were provided by the study authors in response to us contacting them are indicated in Appendix 7.

To enable comparisons of clinical effectiveness among the included interventions, incidence density RRs with their 95% CIs are displayed in forest plots, with the interventions grouped according to their spatial and temporal scales: regional-scale interventions (see Figure 4), local-scale interventions of duration ≤ 12 months (see Figure 5), and local-scale interventions of duration of > 12 months (see Figure 6). Within each forest plot, studies are ordered by effect size (larger effects of interventions relative to comparators are indicated by smaller incidence density RRs). As this is a narrative synthesis, pooled-effect estimates are not displayed in the forest plots. The amount of information available varied considerably among the studies, with some studies providing RRs for more than one intervention scenario or time period (multiple within-study effect estimates are not statistically independent but are included in the forest plots, as pooled-effect estimates are not calculated). The full data used in calculating RRs, including the baseline and intervention catheter-BSI incidence densities for each study, are given in Appendix 7.

FIGURE 4

Incidence density RRs (± 95% CI) for regional-scale interventions (multiple catheters per patient not counted separately in device-days). a, CI not calculable

FIGURE 5

Incidence density RRs (± 95% CI) for local-scale interventions of up to 12 months in duration (multiple catheters per patient not counted separately in device-days). a, Confidence interval not calculable

FIGURE 6

Incidence density RRs (± 95% CI) for local-scale interventions of more than 12 months in duration (multiple catheters per patient not counted separately in device-days unless stated)

Educational interventions were classified either as effective at reducing catheter-BSI incidence density, lacking convincing evidence for effectiveness, or ineffective according to the following criteria:

• Effective The incidence density RR for intervention compared with comparator (baseline) was < 1.0 and the 95% CI of the RR did not include 1.0.
• Ineffective The 95% CI of the incidence density RR for intervention compared with comparator (baseline) included 1.0.
• Lack of convincing evidence for effectiveness Effectiveness was marginal, or serious limitations of the study methodology cast doubt on the reliability of the results (explanations are provided below on a case-by-case basis).

Regional-scale interventions

Five studies^{34,68,83,126,136} investigated regional-scale CQI interventions (Figure 4; for the full data see Appendix 7, Data for forest plot: regional-scale interventions). These studies included the Keystone ICU project,^34,123,124 the ‘CLAB ICU’ project,^83,146, which sought to replicate aspects of the Keystone ICU project in Australia, and the ‘Bacteraemia Zero’ project,⁶⁸ which sought to replicate aspects of the Keystone ICU project in Spain. Three of the studies used uncontrolled before-and-after designs, whereas two (a RCT by Speroff and colleagues¹³⁶ and the non-randomised controlled study by Palomar Martinez and colleagues⁶⁸) included two parallel comparison groups of critical care units (virtual collaborative and toolkit groups¹³⁶ or intervention and control groups⁶⁸). The study by Palomar Martinez and colleagues⁶⁸ reported historical baseline data on infection incidence for 3 years prior to the prospective intervention (2004–6), of which we report only the 2006 data, as these are most directly relevant for comparison with the study period (2007) (for full data see the data extraction form in Appendix 5). All of the regional-scale studies calculated the number of device-days based on presence/absence of vascular catheters, which does not take into account the number of concurrent catheters that a patient may have.

Clinically effective interventions (with unclear risk of bias)

Assuming that effects displayed in the forest plots reflect those of the planned interventions, the CLAB ICU project^83,146 and Keystone ICU project^34,123,124 interventions appear to have been effective in reducing catheter-BSI incidence densities relative to baseline, although both studies had unclear risk of bias (Box 1). The Keystone ICU project achieved a reduction in catheter-BSI incidence density after 3 months, which subsequently persisted through the 36-month intervention monitoring period. The CLAB ICU project achieved a reduction in catheter-BSI incidence density after 6 months, which was mostly sustained until the end of the 18-month intervention monitoring period (statistically significant in quarters 3, 5 and 6) (see Figure 4). These studies both have some relevance to clinical practices for prevention of catheter-BSI in critical care in the NHS and are considered in more detail below.

BOX 1

Regional-scale interventions at unclear risk of bias that appear effective at reducing the incidence density of catheter-BSI CQI programme conducted in 103 critical care units in the USA: the Keystone ICU project (duration of up to 36 months). Targeted (more...)

Clinically ineffective interventions

The two interventions implemented in the RCT by Speroff and colleagues¹³⁶ were clearly not effective at reducing catheter-BSI incidence density, as acknowledged by the study authors (Box 2). During the 18-month study period, catheter-BSI incidence density for the virtual collaborative intervention actually increased relative to baseline (see Figure 4).

BOX 2

Regional-scale interventions that were not effective at reducing the incidence density of catheter-BSI or lack convincing evidence for effectiveness Not effective: CQI programme conducted in 60 critical care units in the USA during 18 months. Hospitals (more...)

Process evaluation conducted by Speroff and colleagues¹³⁶ (discussed below: see Process evaluations, facilitators and barriers) indicated that adoption of intervention components was consistently higher in the virtual collaborative intervention arm than the toolkit approach arm, suggesting that failure of the interventions to reduce catheter-BSI incidence was not simply related to the extent of intervention implementation.

Interventions lacking convincing evidence for effectiveness

The two remaining regional-scale interventions, by Render and colleagues¹²⁶ and Palomar Martinez and colleagues,⁶⁸ appear at first sight to have been effective at reducing the incidence densities of catheter-BSI. However, upon closer inspection the findings of these studies are difficult to interpret.

Published data for the study by Render and colleagues¹²⁶ indicate an incidence density RR below 1.0 but the publication did not provide a CI or sufficient data for us to calculate one. Further data obtained on request from the author (see Appendix 7, Data for forest plot: regional-scale interventions) enabled us to calculate an incidence density RR with a 95% CI, but the CI indicates lack of effectiveness (see Figure 4). On balance, the available information for the study by Render and colleagues¹²⁶ does not provide convincing evidence that the intervention was effective at reducing the incidence density of catheter-BSI (see Box 2).

The results of the Bacteraemia Zero project reported by Palomar Martinez and colleagues⁶⁸ are difficult to interpret because incidence density RRs were significantly lower than 1.0 both for the intervention and control groups of critical care units, with a larger effect (smaller RR) evident in the control group (see Figure 4). The Palomar Martinez study⁶⁸ is notable in that we judged it to be at high risk of selection bias because the control group had higher baseline incidence density of catheter-BSI than the intervention group. We also judged this study to be at high risk of performance bias because staff in control critical care units were able to attend intervention group meetings at which intervention information and materials were disseminated. On balance, this 3-month pilot study by Palomar Martinez and colleagues⁶⁸ does not provide convincing evidence of effectiveness of their CQI programme at reducing the incidence density of catheter-BSI.

Local-scale interventions of up to 12 months in duration

Twelve studies investigated local-scale interventions of up to 12 months' duration. Of these, 10 studies^{50,51,98,103,108,109,122,129,139,144} provided sufficient data for the calculation of incidence density RRs (Figure 5; for the full data, see Appendix 7, Data for forest plot: local-scale interventions of duration up to 12 months). These studies all calculated the number of device-days based on the presence/absence of vascular catheters, which does not distinguish multiple concurrent catheters in a patient. The educational interventions were diverse and included single lectures on practices related to CVC care;^109,122 structured and interactive education modules focusing on hand hygiene and CVC care¹⁴⁴ multimodal education of various types with performance feedback;^{50,51,103,129,139} multimodal education with infection surveillance feedback;¹⁰⁸ and a post-insertion central line care bundle.⁹⁸ Three of these studies each involved a single critical care unit;^50,51,108 three studies involved multiple critical care units and reported results pooled across the units;^98,129,139 and four studies presented results separately for different critical care units.^{103,109,122,144} Lobo and colleagues¹⁰⁹ compared a single lecture in one critical care unit (‘ICU A’) with a tailored continuous educational intervention in another critical care unit (‘ICU B’) located at the same hospital, whereas Higuera and colleagues,¹⁰³ Perez Parra and colleagues¹²² and Zingg and colleagues¹⁴⁴ compared results from different critical care units which had received similar interventions (see Figure 5).

Clinically effective interventions (with unclear or high risk of bias)

Assuming that effects displayed in the forest plots reflect those of the planned interventions, seven of the interventions^{50,51,98,103,129,139,144} would appear to have been effective in reducing the incidence density of catheter-BSI (Box 3), as incidence density RRs were significantly below 1.0 (see Figure 5). The studies by Coopersmith and colleagues⁵⁰ and Warren and colleagues⁵¹ were judged to be at high risk of performance bias because the authors stated that the staff assessing outcomes were not blinded to the intervention. However, it seems unlikely that staff would have been blinded in any of the other studies, although this was not reported. Two studies^50,129 were judged to be at high risk of performance bias for other reasons. Effects of the educational intervention implemented by Coopersmith and colleagues⁵⁰ appear to have been confounded with changes in antibiotic prescribing practice and availability of four-lumen catheters, whereas effects of the educational intervention implemented by Rosenthal and colleagues¹²⁹ overlapped with a separate hand washing intervention. One study by Zingg and colleagues¹⁴⁴ was judged to be at high risk of selection bias, as the study characteristics of the population differed between the baseline and intervention periods.

BOX 3

Local-scale interventions of up to 12 months' duration and at unclear (or, where stated, high) risk of bias that appear effective at reducing the incidence density of catheter-BSI • Multimodal education including self-study, in-services and performance (more...)

In two^103,129 of the seven studies that appeared clinically effective, further caution on interpretation is required. In the study by Rosenthal and colleagues¹²⁹ incidence density RRs were significantly < 1.0 for education or education and performance feedback periods together but not for the performance feedback period alone. These results are difficult to interpret because the education and performance feedback periods were sequential, and performance feedback may not have been independent of the effects of the preceding education. Interpretation should therefore be restricted to the education phase, which itself appeared to be clinically effective, although this was only monitored for 1–2 months (see Figure 5). In the study by Higuera and colleagues¹⁰³ the intervention was effective when data for two critical care units were pooled but found to be effective in only one of the units when they were analysed separately (see Figure 5). It is notable that the medical–surgical critical care unit had a higher baseline incidence density of catheter-BSI per 1000 catheter-days (57.4) than the neurosurgical critical care unit (32.8). These baseline incidence densities and also the baseline catheter-BSI incidence density in the study by Rosenthal and colleagues¹²⁸ (45.9 per 1000 catheter-days) are higher than typically found in European studies.

For these seven studies^{50,51,98,103,129,139,144} that appeared to be effective at reducing incidence of catheter-BSI (albeit with the caveats noted above), statistical significance indicated by the CIs of RRs in Figure 5 is consistent with the primary study publications, which, in all cases, claimed that effects of the interventions at reducing catheter-BSI incidence were statistically significant.

Two^51,139 of the studies that appeared effective at reducing catheter-BSI incidence (see Box 3) reported post-intervention follow-up monitoring, which may provide an indication of the longer-term persistence or attenuation of intervention effectiveness. Warren and colleagues¹³⁹ conducted 10 months of follow-up after a 3-month intervention; and Warren and colleagues⁵¹ conducted 23 months of follow-up after an intervention of about 1 month. Although the RRs based on the whole follow-up period appear encouraging, monthly data provided for the latter study⁵¹ (data extraction form – see Appendix 5) show that incidence densities varied considerably from month to month, and returned to baseline levels within the first 3 months of the 23-month follow-up period.

Clinically ineffective interventions

For two^109,122 of the local-scale short-term studies, the 95% CIs for the RRs indicate that the reduction in catheter-BSI incidence density was not statistically significant (see Figure 5). In contrast, the primary publications for these studies both claimed statistically significant effects of the interventions.^109,122 Lobo and colleagues¹⁰⁹ based their conclusion on incidence data rather than incidence density. Perez Parra and colleagues¹²² stated that they initially conducted a Wilcoxon rank sum test and then to control for (unspecified) confounding effects of external events and (unspecified) secular trends that occurred during the study period they used a Poisson regression approach. The statistical significance reported¹²² varied among the critical care units (overall: p = 0.3; general post surgical: p = 0.05; cardiac post surgical: p = 0.12; medical: p = 0.31) and it is not clear to which of the analytical approaches the p-values refer. On balance, we conclude that these two studies were not effective at reducing catheter-BSI incidence density (see Figure 5 and Box 4) and the claims of statistical significance in the primary publications do not provide sufficient grounds for us to alter our conclusion.

BOX 4

Local-scale interventions of up to 12 months' duration that were not effective at reducing the incidence density of catheter-BSI or lack convincing evidence for effectiveness Not effective: Single lecture (15 minutes with 9 months of follow-up) (more...)

Interventions lacking convincing evidence for effectiveness

The intervention reported by Lobo and colleagues¹⁰⁸ has a RR bordering on statistical non-significance (upper limit of the 95% CI 0.97) and a non-significant RR for the follow-up period suggesting that the intervention was only briefly effective (see Box 4) at reducing the incidence of catheter-BSI (see Figure 5). The publication for this study did not report statistical significance.¹⁰⁸

Local-scale interventions of more than 12 months in duration

Seven studies^{87,94,95,97,110,138} investigated local scale interventions of > 12 months in duration, of which five provided sufficient data for the calculation of incidence density RRs (Figure 6: for the full data see Appendix 7.3, Data for forest plot: local-scale interventions of duration of > 12 months). The studies were all conducted in single critical care units, apart from one which involved three units.⁹⁷ The interventions included multimodal education with slide shows and bedside in-services, repeated for up to 6 years;^94,95 multimodal education with self-study, bedside in-services and performance feedback over 15 months;⁸⁷ a central line bundle deployed over 19 months, including checklist, infection surveillance feedback, performance feedback and catheter supplies cart;⁹⁷ a CQI programme implemented over 3 years, including checklist, infection surveillance feedback and performance feedback;¹¹⁰ and a CQI programme focused on real-time performance feedback over 2 years, including a checklist, supervision of insertions and web-based tutorial with competence assessment.¹³⁸

Three^87,97,138 of these five studies calculated the number of device-days based on presence/absence of vascular catheters, which does not distinguish multiple concurrent catheters in a patient. Eggimann and colleagues⁹⁴ and Longmate and colleagues¹¹⁰ calculated the number of device-days in two ways: based on the presence/absence of vascular catheters; and by counting each concurrent catheter as a separate device-day. In both studies the incidence density RRs were similar for both methods of calculating the device-days (see Figure 6).

Clinically effective interventions (with unclear risk of bias)

Assuming that effects displayed in the forest plots reflect those of the planned interventions, three of the interventions conducted by Eggimann and colleagues,^94,95 Longmate and colleagues¹¹⁰ and Galpern and colleagues⁹⁷ appear effective in reducing the incidence density of catheter-BSI (Box 5), with incidence density RRs significantly lower than 1.0 (see Figure 6). Exceptions were that the RRs were not significantly different from 1.0 on all of the monitoring dates reported by Eggimann and colleagues^94,95 and Longmate and colleagues.¹¹⁰ Data from the Eggimann study^94,95 suggest that the intervention was effective for the initial 3 years of implementation but not consistently so thereafter. Data from the Longmate study¹¹⁰ suggest that the intervention did not become effective at reducing catheter-BSI until the third year of implementation (see Figure 6). The publications reporting these three studies^94,95,97,110 either claimed statistically significant intervention effects^97,110 or did not mention the statistical significance of effects.^94,95

BOX 5

Local-scale interventions of > 12 months’ duration and at unclear risk of bias that appear effective at reducing the incidence density of catheter-BSI • Multimodal education based on 30-minute slide shows and bedside in-services (more...)

The study by Longmate and colleagues¹¹⁰ was judged to be at high risk of performance bias, as the authors stated that the staff assessing outcomes were not blinded. However, as mentioned above, it is unlikely that staff would have been blinded in any of the other studies, although this was not reported. The other studies were judged to be mostly at unclear risk of bias, although (as mentioned in the section on quality assessment above) the Eggimann study⁹⁴ was judged to be at low risk of bias for four of seven bias domains assessed (the other three were judged unclear).

Clinically ineffective interventions

Two^87,138 of the five studies that implemented local-scale interventions of > 12 months' duration, by Coopersmith and colleagues⁸⁷ and Wall and colleagues,¹³⁸ were not effective at reducing catheter-BSI incidence density (Box 6), as the incidence density RRs were not significantly different from 1.0 (see Figure 6). Coopersmith and colleagues⁸⁷ acknowledged that effects of their intervention were not statistically significant. Among possible reasons for the lack of success of the intervention, Coopersmith and colleagues⁸⁷ suggested that that diffuseness of the educational message may have been a contributory factor and that ideally didactic education should be specifically targeted to support best clinical practices. Wall and colleagues¹³⁸ did not report the statistical significance of the effectiveness of their real-time process feedback approach for reducing catheter-BSI. The number of infections appeared to decrease dramatically (from 25 per 24 months to 6 per 24 months),¹³⁸ but there was also a marked decrease in the number of CVC-days during the study, which explains the lack of statistical significance when intervention effects are analysed in terms of the incidence density of catheter-BSI.

BOX 6

Local-scale interventions of more than 12 months' duration that were not effective at reducing the incidence density of catheter-BSI • Multimodal education on CVC care and unspecified topics, including self-study, in-services and performance feedback, (more...)

Overview of intervention effects on catheter-BSI incidence density

Assuming that the incidence density RRs reflect those of the planned interventions, then a range of different types of educational intervention would appear to have been effective at reducing catheter-BSI incidence in critical care units (Boxes 1, 3 and 5). A key proviso is that some studies were judged to be at high risk of specific types of bias, but risk of bias was generally unclear and it is not possible to objectively classify the studies on bias risk. The interventions that appeared effective, subject to the caveats above, include regional-scale CQI programmes^34,83 or a local-scale CQI programme;¹¹⁰ local-scale multimodal education with or without performance feedback and infection surveillance feedback;^{50,51,94,103,129,139,144} a catheter insertion bundle;⁹⁷ and a catheter ongoing care bundle.⁹⁸ However, single lectures on CVC care and infection prevention were not effective as a means of reducing catheter-BSI incidence densities.^109,122 There is no clear evidence to suggest that regional-scale studies were more effective than local-scale studies, or that short-term studies (up to 12 months' duration) had different effectiveness than long-term studies (exceeding 12 months' duration). All three groups of studies (regional scale, local scale short-term and local-scale long-term) contain examples of effective interventions (Boxes 1, 3 and 5) and ineffective interventions (Boxes 2, 4 and 6). The studies included in the systematic review provide no clear evidence that performance feedback and/or infection surveillance feedback were influential in achieving effectiveness. Among the 12 interventions classed as clinically effective, eight (67%) included performance feedback,^{50,51,83,97,98,103,110,129} four (33%) included infection surveillance feedback,^34,83,97,110 three (25%) included both types of feedback^83,97,110 and two (17%) did not include either type of feedback.^94,144 Among the eight interventions that were not classed as clinically effective, the respective proportions were similar: four (50%) included performance feedback,^{109,126,136,138} three (38%) included infection surveillance feedback,^108,109,126 two (25%) included both types of feedback^109,126 and one (13%) did not include either type of feedback.¹²² Apart from single lectures being ineffective, there appears to be no clear evidence that interventions which included components beyond education (e.g. provision of antiseptic or a catheter supplies cart) were more or less effective than interventions that consisted of education alone.

Most studies did not separately count multiple vascular catheters per patient when calculating the number of device-days. In two studies that compared different approaches for calculating the number of device-days, the calculation approach used did not appear to influence the incidence density RR for catheter-BSI.

Mortality

Five studies^{94,103,110,139,144} reported mortality as an outcome but they did not specifically report mortality due to catheter-BSI or its sequelae.

Only one of these studies¹¹⁰ reported mortality specifically for critical care patients with CVCs (those patients in the critical care unit for > 2 days with a CVC for at least part of their critical care stay). Mortality rates significantly decreased during the study period, being 21.2% during the baseline period, and 20.9% and 16% during years 3 and 4 of the intervention, respectively (year 3 vs. 1, p = 0.328; year 4 vs. 1, p = 0.013).

Three^94,103,144 of the five studies reported unadjusted mortality rates for critical care patients. Of these, two reported that there were no statistically significant differences between the baseline and intervention periods.^94,144 An exception was that mortality due to cardiac arrest was significantly more frequent during the intervention period of one study⁹⁴ (p < 0.05). The third study¹⁰³ reported statistically significantly lower rates of unadjusted mortality per 1000 critical care unit discharges during the intervention period than the baseline period (64/132 = 48.5% and 111/338 = 32.8%), respectively [RR = 0.68 (95% CI 0.50 to 0.91), p = 0.01].

The remaining study¹³⁹ reported that mortality for an ambiguous population (‘in-hospital mortality rate for catheterised patients’) did not differ significantly between the baseline and intervention periods.

In summary, insufficient data are available for us to draw any firm conclusions about the effects of the educational interventions, or the effects of catheter-BSI, on rates of mortality among patients in critical care units.

Length of stay

Seven studies^{50,94,109,110,117,139,144} reported LOS. One of these studies¹¹⁰ explicitly stated that their LOS data were for critical care unit patients who had vascular catheters. One study¹⁴⁴ compared LOS in patients with and without CRBSI. For the remaining five studies, LOS data appear to refer to all critical care unit patients, not limited to those with vascular catheters.

Longmate and colleagues¹¹⁰ reported both mean and median lengths of stay for patients who were in the critical care unit for > 2 days and had a CVC for at least part of their critical care stay. Mean (± SD) lengths of stay significantly increased [5.4 ± 3.9 days during year 1 (baseline period), 5.3 ± 3.7 days during intervention year 3, and 5.9 ± 3.7 days during intervention year 4] (differences from baseline, p > 0.05). The corresponding median [interquartile range (IQR)] lengths of stay decreased, and were, respectively, 9.7 (4–20) days, 8.9 (4–13) days and 6.0 (3–11) days (differences from baseline, p < 0.05).

Zingg and colleagues¹⁴⁴ reported median overall LOS for patients in five critical care units, during a 4-month baseline and 5-month intervention period. LOS was statistically significantly longer during the intervention period. The median (IQR) LOS for baseline and intervention periods respectively were 3 (2–7) days and 4 (2–9) days – difference, p < 0.001. Corresponding mean LOS during these periods were 5.9 days and 7.5 days, respectively. Zingg and colleagues also reported that median (IQR) LOS was 15.5 (10–25) days in patients with CRBSI and 5 (3–12) days in patients without CRBSI (difference reported as 10.5 days).

In summary, two studies^110,144 reported length of critical care unit stay in relevant populations of patients with CVCs. One study,¹⁴⁴ a 9-month education intervention by Zingg and Colleagues, reported significantly longer duration of stay during the intervention period, whereas the other, a 4-year continuous QI programme by Longmate and colleagues,¹¹⁰ found significantly shorter duration of stay in the intervention period. In addition, Zingg and colleagues¹⁴⁴ reported that the median LOS for critical care patients with CRBSI was 10.5 days longer than for critical care unit patients without CRBSI (reported as 15.5 and 5 days, respectively).

Attitudes

One of the included studies by Sherertz and colleagues¹³⁵ reported staff attitudes as a quantitative outcome. The proportions of postgraduate year 1 physicians-in-training who perceived a need for povidone–iodine, gloves, gowns and full sterile drapes increased significantly following a formal one-day education course (full data are in the data extraction form – see Appendix 5). However, incidence density RRs could not be calculated for this study¹³⁵ so its clinical effectiveness at reducing catheter-BSI incidence density is unclear.

Four studies^{68,83,110,122} mentioned qualitative observations relating to the attitudes of critical care staff towards evidence-based infection prevention practices. Burrell and colleagues⁸³ observed that some clinicians considered the incidence of CLAB in New South Wales to be low and doubted the value of the project, as existing Australian practice was felt to be equal to or better than the methods informing the project. Some clinicians doubted the evidence even with supportive CDC guidelines. Hat-wearing was a contentious element of the physician bundle: clinicians cited lack of evidence for hat use and four critical care units elected to omit their use as standard practice for CVC insertion.⁸³ The remaining studies provided only brief mention of staff attitudes. Longmate and colleagues¹¹⁰ reported that two consultant clinicians doubted the need for full aseptic technique for CVC insertion, although this was resolved after evidence sharing. Perez Parra and colleagues¹²² reported that staff incorrectly assumed that small drapes were sufficient for catheter-BSI prevention, although it is not clear whether they were referring to the baseline or intervention period. Palomar Martinez and colleagues⁶⁸ reported that some practitioners expressed dissatisfaction with chlorhexidine for skin antisepsis and doubted its effectiveness.

Although limited in detail and lacking quantitative analysis, these observations of the attitudes of critical care staff highlight that staff may be liable to question the need for evidence-based infection prevention practices, even when presented with supporting evidence.

Knowledge

Two^50,122 of the included studies indirectly reported improvements in knowledge, expressed as changes in the proportion of staff with correct test scores⁵⁰ or the percentage of correctly answered questions.¹²² In the latter study, the test questions most often answered incorrectly (fewer than 50% correct) were those about the need for full sterile barriers during CVC insertion and on the choice of antiseptic for skin disinfection.¹²² These studies suggest that educational interventions can improve critical care staff knowledge of infection prevention practices but they did not report the types of knowledge gained through participation in the interventions.

Compliance

Compliance with evidence-based practices for preventing catheter-BSI was reported in 19 out of the 24 included studies.

Compliance with hand hygiene

Seven studies reported compliance with hand hygiene behaviour (Table 22). They all reported improvements in compliance relative to baseline, although these were not statistically significant in all cases. In most of the studies hand hygiene compliance rates did not reach 100% during educational interventions. In one study, the change was modest (final compliance 30%) and non-significant, possibly because hand hygiene was not a specific target of the education.⁸⁷

TABLE 22

Compliance with hand hygiene during study baseline and intervention periods

Five^{103,108,109,130,144} of the studies that achieved statistically significant improvements in compliance had specified hand hygiene as a main^108,109,130 or joint^103,144 component of their education. High baseline variability in compliance rates is notable, both within and between the studies. For example, Lobo and colleagues¹⁰⁸ found baseline compliance with hand hygiene at CVC insertion was already 100% but compliance with hand hygiene before line manipulation was only 5%. Overall, these findings suggest that hand hygiene behaviour in relation to the insertion and management of CVCs is complex and variable, and can differ considerably between the stages of intravascular catheter site preparation, insertion and ongoing management. Owing to the relatively small number of studies that reported hand hygiene compliance it is unclear whether the degree of compliance with hand hygiene practices had any bearing on the effectiveness of the interventions for preventing catheter-BSI.

Compliance with barrier precautions

None of the seven studies^{87,108,109,126,135,138} that measured compliance with barrier precautions (Table 23) had specified this as a target behaviour change, although most of the studies included an element of education about sterile barrier precautions in their interventions. Compliance with barrier precautions was highly variable at baseline, ranging from 0% to 100%. Final compliance with barrier precautions after study interventions ranged from 65% to 100%, indicating improvement, but this was reported to be statistically significant for only two out of eight comparisons.

TABLE 23

Compliance with barrier precautions during study baseline and intervention periods

The studies appeared to differ in their ability to detect statistically significant changes in compliance. A 30% increase in compliance with maximal sterile barrier use was not significant (p = 0.29) in one study,⁸⁶ whereas a 21% increase in compliance with sterile drape use was significant (p < 0.001) in another study.¹³⁵ Wall and colleagues¹³⁸ reported that a decline in compliance with maximal barrier precautions during the intervention period was caused by lack of use of patient drapes. Compliance with maximal barrier precautions improved after the team purchased new sterile kits pre-packaged with drapes, and confirmed providers had completed a tutorial. Wall and colleagues¹³⁸ also mentioned that use of the femoral site, compared with other insertion sites, was associated with statistically significant lower compliance with hand washing, chlorhexidine skin antisepsis and maximal barrier precautions.

Compliance with dressing management

None of the six studies that measured compliance with dressing management (Table 24) explicitly specified this was a target behaviour, although dressing care was clearly stated as a topic in the educational interventions of the studies by Coopersmith and colleagues,⁸⁷ Higuera and colleagues¹⁰³ and Lobo and colleagues.¹⁰⁹ Compliance with various aspects of CVC dressing care ranged from 11% to 84% at baseline and improved to 21% to 97% after interventions, with the improvements all being reported to be statistically significant, although compliance with correct dating of CVC dressings reached only 21% after the intervention by Coopersmith and colleagues.⁸⁷ In the study by Rosenthal and colleagues,¹²⁹ compliance with placing gauze dressings and with checking the condition of dressings both increased by a greater degree following performance feedback than following education alone. However, as these intervention components were implemented sequentially, the effects of performance feedback may not have been independent of the education.

TABLE 24

Compliance with dressing management during study baseline and intervention periods

Compliance with skin antisepsis prior to catheter insertion

Skin antisepsis prior to CVC insertion was a common element of the education in many of the studies included in the systematic review, but no studies specified explicitly that it was a target behaviour. Baseline rates of compliance with skin antisepsis ranged from 9% to 57% in the three studies that reported this outcome, and in all cases interventions resulted in improvements, with final compliance ranging from 82% to 100%. The change was statistically significant in one study¹⁰⁸ but significance was not reported in the other two studies^126,138 (see Table 25).

TABLE 25

Compliance with other activities during study baseline and intervention periods

Compliance with hub disinfection and catheter set dating

Compliance rates for line protection and hub disinfection were reported in two studies^108,109 and ranged from 33% to 69% at baseline and were improved by interventions, with final compliance in the range 82% to 98% (all improvements were reported statistically significant). However, there was a notable difference between two studies in compliance with the dating of intravenous administration sets, although in both cases increases in compliance occurred, which were statistically significant. Baseline compliance was only 0.57% and initially fell to 0% after education in the study by Rosenthal and colleagues¹²⁹ but then reached 74% after a phase of performance feedback. In the study by Higuera and colleagues,¹⁰³ compliance with set dating rose from 40.69% to 93.85% (see Table 25).

Compliance with overall bundles

Three studies^83,98,110 reported compliance with overall care bundles (Table 25). These were ‘patient’ and ‘clinician’ bundles⁸² a CVC insertion bundle¹¹⁰ and a post-insertion care bundle.⁹⁸ Compliance improved in the studies by Burrell and colleagues⁸³ and Longmate and colleagues¹¹⁰ but was reported to be statistically significant only in the former study. The intervention by Guerin and colleagues⁹⁸ did not appear to affect compliance with the CVC post-insertion care bundle. It is notable that compliance with all four bundles was already high at baseline, ranging from 74% to 94%, and improvements appeared modest (increases of only 7% and 11% in the Burrell study⁸³ despite being statistically significant, and approximately 20% in the Longmate study¹¹⁰).

Burrell and colleagues⁸³ demonstrated relationships between compliance with the two care bundles and the effectiveness of their CQI intervention at preventing catheter-BSI (central line-associated bacteraemia). Risk of catheter-BSI was reduced if insertion was conducted by physicians compliant in both bundles [RR = 0.5 (95% CI 0.4 to 0.8); p = 0.004], but risk was increased if insertion was conducted by physicians not compliant with the clinician bundle [RR = 1.62 (95% CI 1.1 to 2.4); p = 0.018. Most (94.0%) cases of non-compliance with the clinician bundle were due to failure to wear a hat, mask and eyewear.

Longmate and colleagues¹¹⁰ reported that all-or-nothing compliance with the CVC insertion bundle component of their CQI programme fluctuated between 80% and 100% during an 18-month period. The authors suggested that low initial compliance was attributed to a lack of checklist stickers early in the period; improved compliance later may have been due to introduction of a CVC insertion pack and the creation of subteam nurses to ‘own’ CVC processes, and also to greater scrutiny and follow up of episodes of incomplete compliance.¹¹⁰

In addition to the studies listed in Table 25, Speroff and colleagues¹³⁶ provided extensive data on the adoption of various components of flexible QI interventions in critical care units of hospitals that had been randomised to toolkit-based and virtual collaborative QI approaches (full data are given in the data extraction form: see Appendix 5). Adoption of intervention components was consistently higher in virtual collaborative hospitals than toolkit hospitals. However, as mentioned above, neither of these intervention approaches was successful in reducing catheter-BSI incidence density relative to baseline. Risk of catheter-BSI appeared higher under the virtual collaborative (see Figure 4), which had the more frequent adoption of intervention tools and strategies, suggesting that failure to reduce catheter-BSI incidence density was not simply related to the extent of intervention implementation.

Further data on compliance with evidence-based practices were reported by Coopersmith and colleagues,⁵⁰ DuBose and colleagues,⁹³ Lobo and colleagues,¹⁰⁹ Palomar Martinez and colleagues,⁶⁸ and Render and colleagues.¹²⁶ These data (see Appendix 5) are not discussed here as they are based on very small or unclear sample sizes, or it is unclear to which study periods they refer.

Summary of compliance

Overall, nearly all of the studies that reported compliance with evidence-based infection prevention practices reported improvements relative to the baseline period, although the uncontrolled studies cannot definitively exclude an influence of secular trends. Most of the information on compliance relates to hand hygiene, barrier precautions and CVC dressing care. The studies are difficult to compare, however, as in some studies small changes in compliance were reported to be statistically significant, whereas in other studies large changes were reported not statistically significant. Interventions targeting specific behaviours appear more likely to improve compliance but this is difficult to assess critically as target behaviours were not always clearly specified in the primary studies. Baseline compliance rates varied considerably between studies and for hand hygiene they varied markedly between the different stages of CVC site preparation, insertion and ongoing care. It is not clear from these data whether compliance with evidence-based practices was an important mediator of intervention effectiveness at preventing catheter-BSI, as baseline compliance rates were often already high. An exception is the study by Burrell and colleagues,⁸³ which found that compliance with two care bundles significantly reduced the risk of catheter-BSI.

Other secondary outcomes

Our systematic review protocol (see Appendix 1) specified two secondary outcomes that we would assess if reported in the primary studies: (1) the reaction of critical care staff to education and (2) critical care staff practical skills in relation to infection prevention. However, none of the 24 studies included in the systematic review reported these outcomes.

Process evaluations, facilitators and barriers

In addition to the assessments of attitudes, knowledge and compliance reported above, six studies^{34,68,83,110,126,136} provided qualitative observations relevant to understanding intervention processes, facilitators and barriers, although none of these studies carried out a full process evaluation.

Three of the CQI studies^83,110,136 identified a lack of adequate infrastructure to support data collection and dissemination as being a barrier to successful implementation of the interventions. Burrell and colleagues⁸³ stated that reliable baseline data did not exist prior to the project owing to variable reporting mechanisms. Some critical care units were hesitant to accept previously reported rates, and inadequate staffing rates in some critical care units impacted on data capture rates. Difficulties were also encountered regarding data collection and associated information technology, such that the project team resorted to hard copy receipt of checklists. The lack of a continuous and sustainable data collection system involving cross-specialty collaboration was seen as a serious risk to sustainability of the CLAB ICU project principles. Speroff and colleagues¹³⁶ commented that the lack of appropriate infrastructure to support data-driven QI was a significant barrier and that systematic standardised data collection was initially lacking in many of the study hospitals. Early effort was therefore needed to deploy a system-wide standardised infection control database registry. Longmate and colleagues¹¹⁰ reported that investigators were initially unable to reach agreement on a system – which had full support of both clinicians and data analysts – for collecting process measurements.

Three studies^34,68,110 reported difficulties around the use of chlorhexidine for skin antisepsis. These related to difficulty in obtaining supplies (implying uneven compliance across critical care units)^34,68 and the fact that chlorhexidine is colourless, making the skin area prepared with antiseptic difficult to see.^68,110 In one study¹¹⁰ it was agreed that povidone–iodine could be used to colour the skin, followed by chlorhexidine.

Two studies^68,136 commented that their8 interventions provided insufficient time for some components to be implemented. Speroff and colleagues¹³⁶ reported that implementation of checklists was slow, suggesting that beneficial translation of desired changes may take more than 18 months to achieve. Palomar Martinez and colleagues⁶⁸ reported that none of the critical care units was able to implement catheter equipment carts within the 3-month duration of their pilot study.

Other aspects of intervention process evaluation were identified in specific studies:

Burrell and colleagues⁸³ reported difficulty in ensuring application of, and adherence to, infection surveillance definitions. They also commented that the CLAB ICU project methodology was based on a single successful collaborative (the Keystone ICU project) without a detailed analysis of all available evidence, which allowed criticism of methodology to be an excuse for non-compliance.

Palomar Martinez and colleagues⁶⁸ reported that critical care staff were reluctant to take a test as part of the training for the CQI intervention, as the test was not mandatory, results were not anonymous and there was no credit given for participation in the training.

Pronovost and colleagues³⁴ reported that the Keystone ICU project intervention was modestly more effective in small hospitals, with an incidence rate ratio of 0.97 (95% CI 0.96 to 0.99, p < 0.001) for each 100-bed decrease in the size of the hospital. Although Pronovost and colleagues³⁴ did not conduct a detailed process evaluation during the Keystone ICU project, Dixon-Woods and colleagues¹⁴⁹ developed an ex-post theory of the processes that contributed to the project's success at preventing catheter-BSI. Although not noted by Dixon-Woods and colleagues,¹⁴⁹ an important difference between the Keystone ICU project³⁴ and other regional-scale CQI approaches that we reviewed^{68,83,110,136} could be that the Keystone ICU project was based on an established data collection system.

Render and colleagues¹²⁶ provided a table of facilitators and barriers in their publication but it is unclear whether this was based on quantitative evidence. The study authors reported that after 6 months the project leader and project co-ordinator reviewed their detailed notes from the monthly reporting meetings to independently identify themes contributing to or delaying project success. The list of barriers and facilitators was then validated by the project leaders. However, the validation process was not explained.