Participants and sample collection

Phase 1 evaluation was conducted in the Oxford University Hospitals NHS Trust. All samples collected were initially sent to the service microbiology laboratory for faecal culture and/or C. difficile toxin testing by hospital-based doctors or GPs as a result of a suspected enteric infection. Consecutive samples positive for one of the three key pathogens being tested routinely (C. coli, C. jejuni and S. enterica) were put aside by routine microbiology staff rather than being sent for discard, as were samples that had been tested for C. difficile by toxin EIA in the service microbiology laboratory and that were negative for all pathogens. The only exclusion criterion was insufficient sample remaining after standard microbiological testing. During the period of the study, the service microbiology laboratory in the Oxford University Hospitals used an EIA to identify C. difficile manufactured by Meridian Bioscience, Inc. (Cincinnati, OH, USA). A large study conducted in 2011 independently of the manufacturer⁵ demonstrated that this EIA test has particularly poor sensitivity compared with the gold standard cell cytotoxicity assay (69.2%), considerably lower than an alternative EIA test (82.3%) which has been used in Oxford University Hospitals since April 2012 as part of a ‘two-step’ testing algorithm mandated by the UK Department of Health.²⁵ There are two reasons why EIA tests have been used historically by many trusts, and continue to be used within the ‘two-step’ testing algorithm to detect C. difficile: first, their turnaround time is shorter (≈ 1 day compared with 3 days for the cell cytotoxicity assay) and, second, their cost (consumables and staff) is lower, particularly important given the large number of tests performed (≈ 10,000 tests/year in Oxford University Hospitals). Separately to the MassCode study, all EIA-positive C. difficile samples were cultured in the parallel research laboratory under a separate research protocol. Samples positive for C. difficile on both EIA and culture were retrieved from this sample collection as reference positives for use in the MassCode study. As norovirus testing was not carried out by the microbiology laboratory unless an outbreak was suspected, qPCR-positive norovirus samples were obtained through a separate investigation conducted by a NIHR clinical fellow.

A research assistant not involved in the MassCode study therefore collected samples from the three sources (culture-positive C. difficile samples from the research laboratory, PCR-positive norovirus samples from a different study in the research laboratory and samples positive for Campylobacter spp., Salmonella spp. or negative for all pathogens from the service microbiology laboratory). The independent research assistant assigned each sample to one of 1000 pre-generated study numbers at random and maintained a list of which sample corresponded to which study number and what pathogen (if any) had been identified by the service microbiology laboratory. The research assistant produced a blinded random order of samples from the various sources for testing using the MassCode assay from this list (so that not all C. difficile- and norovirus-positive samples were processed in the same batches). As numbers of S. enterica-positive samples were lower than predicted, additional S. enterica-positive samples were sourced from Leeds, sent to the research assistant not involved with the MassCode study, assigned anonymous study numbers and periodically inserted into the workflow as for C. difficile and norovirus-positive samples. All samples were stored at 4 °C prior to processing.

All samples used in the phase 1 blinded evaluation were collected independently to those samples used to determine the extraction protocol (see Chapter 3).

Reference standard

Initial diagnosis of the target faecal pathogens was performed in accordance with Public Health England guidelines in the service microbiology laboratory. Approximately 1 g of faecal sample was inoculated into selenite broth and the broth inoculated onto xylose lysine deoxycholate (XLD) agar for culture of Salmonella spp. and Shigella spp., sorbitol MacConkey (SORB) agar for culture of E. coli O157 and Campylobacter-free blood (CAMP) agar for culture of Campylobacter spp. CAMP agar was incubated microaerophilically at 42 °C for 48 hours, XLD, SORB and the selenite broth, for culture of Salmonella spp. was incubated at 37 °C for 24 hours. After the 24-hour period, the selenite broth was inoculated onto chromogenic agar (SALM) for culture of Salmonella spp. This was incubated for a further 24 hours at 37 °C. Suspect colonies on XLD or SALM were inoculated onto analytical profile index (API^®) 10S, for identification of Salmonella spp., a Columbia agar slope for slide agglutination tests, and a MacConkey agar purity plate. If C. difficile infection was suspected, samples were subject to EIA testing for toxins A and B. Subsequent serological and sensitivity testing was performed as required for each organism identified. Throughout all reference standard and index tests, normal aseptic microbiological laboratory working practices were followed. Campylobacter was identified only to the species level, i.e. C. jejuni and C. coli were not distinguished by the reference standard testing.

Reference standard testing was performed by trainees and state-registered biomedical scientists (BMSs), but all results were confirmed by an experienced state-registered BMS before being passed onto a doctor. As reference tests were carried out before the MassCode assay was run, staff performing the reference assays did not know the results of the MassCode assay.

Blinded investigation

The full SOP for sample preparation and processing is detailed elsewhere (see Appendix 1). A total of 948 clinical samples were collected and extracted using the optimised protocol. This included 200 Campylobacter spp., 199 C. difficile, 60 S. enterica, 199 norovirus and 295 negative samples (some samples had more than one pathogen), compared with targets of 200, 200, 100, 200 and 300, respectively. Insufficient S. enterica-positive samples accrued during the period of the study. Samples were reverse transcribed and stored at −20 °C prior to amplification with the MassCode 13-plex primer mix. Following amplification, samples were cleaned according to the MassCode SOP and analysed by mass spectrometry (MS). All extraction batches included a water extraction as a control and all MS plates included MassCode calibrant controls.

All MassCode assays were conducted by a postdoctoral fellow and a research assistant who had received training on the MassCode assay from Agilent Technologies, and had jointly conducted the pre-phase 1 evaluations (see Chapter 2). The output of the MassCode assay was a number, indicating that the sample was positive on both PCR products for a pathogen (positive), near threshold values or positive on one product and near threshold value for the second product (indeterminate), or was not positive on either (negative). No expert interpretation was, therefore, required to read the MassCode outputs, which were also exported electronically from the spectrometer. The cut-offs defining positivity were set by Agilent Technologies, based on their in-house development and were not altered for this evaluation.

Researchers conducting MassCode assays were blinded to results of the reference test and all other clinical information; samples were identified only by their unique study number (assigned by a different laboratory researcher not involved with the MassCode study), i.e. were completely anonymised with regards to patient and microbiological characteristics (what pathogen, if any, had had been isolated).

MassCode results were used only for the research study and were not returned for patient management.

Sample size

In phase 1 testing, 200 positive detections of C. difficile, norovirus and Campylobacter spp. with reference standard culture/toxin detection and new PCR-based tests would produce a 95% CI around sensitivities of 90% and 96% of ± 4.2% and ± 2.7%, respectively. Testing 100 positive isolations of Salmonella spp. would produce a 95% CI around a sensitivity of 90% of approximately ± 5.9%. Samples positive for each pathogen would act as negative controls for other pathogens, together with 300 samples negative for all pathogens (total 1000 samples to be tested in Oxford and Leeds hospitals).

Analysis

Analysis of the MassCode assay results was conducted by an independent member of the team. All microbiological results were extracted from the microbiology database to confirm the reference standard test results, and identify any co-infections with other pathogens in the 13-plex MassCode panel which had not been part of original sample retrieval procedures. Sensitivity and specificity were calculated compared with the microbiological reference standard for each of the main organisms (primary reference standard), counting identification of either Campylobacter species as correct (as reference microbiological testing did not speciate). Primary analysis did not count indeterminate results (either forward or reverse primer above pre-specified threshold, but not both) as a positive: secondary analysis included these indeterminates as positive. Uncertainty was quantified using 95% CIs. All analysis were conducted in Stata 11.2 (StataCorp LP, College Station, TX, USA).

Samples which were either false negative for any of the five key pathogens or false positive for any of the 11 pathogens on MassCode were then retested using single qPCRs. This testing was not done blinded as the researchers needed to know which qPCR to run. The qPCR assays used are described elsewhere (see Chapter 2 and Appendix 1). Each qPCR assay was run in duplicate and any discrepancies confirmed by additional qPCRs.

As several unexpected positives were confirmed by qPCR results (implying the target pathogen was genuinely present in the sample but missed in the original microbiology work-up), and as some missed positives could not be detected in the original sample, even using single qPCR, a secondary analysis was carried out in which the MassCode-positive results were compared with a combined standard of reference microbiological assay plus single qPCR testing. In this secondary reference standard, samples positive on standard reference microbiology but negative on qPCR were considered negative, and samples negative on standard reference microbiology but positive on qPCR were considered positive. Possible explanations for the former are sample degradation or labelling errors. This secondary reference standard does not reflect a gold standard, but rather a standard whereby single qPCR demonstrates that, respectively, either it is unrealistic to expect the multiplex MassCode assay to detect the nucleic acid target (because it cannot be detected even with a single qPCR) or that the target was present and should have been detected by MassCode even if not identified on original microbiological testing. Effectively this standard addresses the impact on diagnostic capability of multiplexing the MassCode assay rather than running multiple single qPCRs.

Phase 2

Analysis of sensitivity/specificity of MassCode compared with standard reference methods was to be conducted at the end of phase 1 in each trust.

A small steering group was set up to review results of phase 1 (in each hospital and overall) in order to determine whether or not phase 2 should proceed. Criteria for not moving to phase 2 were pre-specified as extremely poor estimated sensitivity for detecting any of the key organisms (under 75%) such that the test would not be likely to be adopted in routine NHS practice. The choice of a threshold of 75% was based on estimated sensitivity of the currently widely used ELISAs for detecting C. difficile. The steering group comprised investigators (DWC, TEAP, MHW and ASW) plus two independent researchers – Professor Ajit Lalvani (Imperial College, expertise in development, assessment and implementation of interferon gamma release assays) and (as chairperson) Dr Christine McCartney (Director of the Health Protection Agency Regional Microbiology Network) and one independent member from outside academia, Ms Katherine Innes Ker, who has extensive commercial experience and will also represent the patient community. Inclusion of investigators and independent researchers in one oversight committee was considered appropriate because there was no patient management which could be influenced by knowledge of results to date, as no results were returned for clinical care.

Sequencing Cryptosporidium product

Single SYBR Green PCR results for Cryptosporidium, performed as part of the phase 1 analysis, suggested that the primer pair targeting Cryptosporidium was resulting in amplification of non-specific targets. To investigate, sequencing of the Cryptosporidium PCR product was performed after the primary analysis of sensitivity and specificity had been conducted. Eight samples were chosen for sequencing, including Cryptosporidium Positive Template Control, C. parvum oocysts, samples found to be positive for Cryptosporidium spp. by reference standard testing, and samples negative/not tested by reference standard testing and found to be positive by MassCode. Extracted samples were subject to conventional PCR following the MassCode SOP (see Appendix 1), but with Cryptosporidium spp. primers in singleplex; forward (5′-GAGGTAGTGACAAGAAATAACAATACAGG-3′) and reverse (5′-CTGCTTTAAGCACTCTAATTTTCTCAAAG-3′) primers were both used at 250 nm. Both primers were designed by Agilent, targeted small subunit ribosomal RNA (SSU rRNA) and were included in the MassCode 13-plex panel.

Following amplification, the PCR product was cleaned to remove unused primers and other ingredients. AMPure XP beads (Agencourt, Beckman Coulter, High Wycombe, UK) were added in a ratio of 1.8 : 1 (18 µl of beads added to 10 µl of PCR product), mixed well and incubated for 3 minutes at room temperature. The beads were separated from the supernatant by placing sample tubes on a magnetic stand and the supernatant removed and discarded. The beads were then washed twice with 200 µl of 70% molecular-grade ethanol. All remaining ethanol was removed and the beads left to air dry for 15 minutes. DNA was eluted in 40 µl of 1 × TE [tris-ethylenediaminetetraacetic acid (EDTA)] buffer by adding the buffer to each sample, mixing well, and incubating at room temperature for 2 minutes. Finally, the beads were separated from the eluate by placing the samples on a magnetic stand and the eluate transferred to new tubes.

A sequencing reaction was performed on the cleaned PCR product using the Big Dye Terminator Cycle Sequencing Kit (Life Technologies). The reaction comprised 1.75 µl of 5 × Sequencing Buffer, 0.5 µl of Big Dye, 0.5 µl of either the forward or reverse primer,and 8.75 µl of molecular-grade water per sample. A total volume of 2 µl of each sample was added to the reaction mix, and each sample was subject to a sequencing reaction with both the forward and reverse primer. Cycling conditions were as follows:

Post-sequencing reaction products were cleaned using CleanSEQ beads (Agencourt). A volume of 10 µl of CleanSEQ beads was added to 10 µl of sequencing reaction product and mixed well. A volume of 42 µl of 85% molecular-grade ethanol was added to each bead sample mixture and, again, mixed well. Sample mixtures were placed on a magnetic stand and incubated for 3 minutes at room temperature to allow the beads to separate from the supernatant. The supernatant was then removed and discarded. The beads were subsequently washed twice with 100 µl of 85% ethanol. Once all ethanol was removed, the beads were air dried for 15 minutes and the DNA eluted from the beads in 80 µl of 0.05 mM EDTA.

The final eluate was sequenced by Applied Biosystems 3730 DNA Analyser (Life Technologies). Forward and reverse sequences for each sample were aligned in Geneious v5.6.6 (Biomatters Ltd, Auckland, New Zealand) and the consensus sequences submitted to Basic Local Alignment Search Tool (BLAST; NCBI) for identification.

Sensitivity and specificity for the main organisms

Sensitivities of each organism varied for the MassCode assay (Figures 6–9 and Table 11), ranging from 43% to 94%. Including indeterminate sample calls (where the MassCode result is near threshold values for a positive call) increased sensitivities only very slightly to 48–96%. Specificities for the MassCode assay were 95–98% or 89–96% including indeterminates. Including qPCR results also led to an increase in sensitivity and specificity, which ranged from 60% to 95% and from 97% to 100%, respectively.

FIGURE 6

MassCode assay sensitivity and specificity for Campylobacter spp. excluding (a) and including (b) qPCR results. Sensitivity = 100 × called positive/true positive; Specificity = 100 × called (more...)

FIGURE 9

MassCode assay sensitivity and specificity for S. enterica excluding (a) and including (b) qPCR results. Sensitivity = 100 × called positive/true positive; Specificity = 100 × called (more...)

TABLE 11

Cross-tabulation of MassCode assay vs. microbiology results

FIGURE 7

MassCode assay sensitivity and specificity for C. difficile excluding (a) and including (b) qPCR results. Sensitivity = 100 × called positive/true positive; Specificity = 100 × called (more...)

FIGURE 8

MassCode assay sensitivity and specificity for norovirus excluding (a) and including (b) qPCR results. Sensitivity = 100 × called positive/true positive; Specificity = 100 × called (more...)

The best-performing organism was C. difficile, although Campylobacter spp. and norovirus also had sensitivities and specificities well above the 75% threshold required by the MassCode protocol in order to proceed to phase 2, with the lower limits of the 95% CI exceeding 83% (sensitivity) and 92% (specificity). However, the sensitivity of S. enterica remained well below this threshold; even including qPCR results the upper limit of the 95% CI around the estimated sensitivity of 60% was just below 75%.

Inspection of the copy numbers returned by qPCR for missed positive samples revealed that 59% of samples positive for Campylobacter spp., 77% of C. difficile-positive samples and 89% of norovirus-positive samples had copy numbers < 100, the previously defined limit of detection. For S. enterica, 61% of the missed positives that were recovered by qPCR had copy numbers < 10, suggesting very low nucleic acid yields.

Tables 12a-c show more detail about the discrepancies between MassCode positives and reference microbiology. C. difficile was the most accurate target, with all but one of the unexpected positives confirmed positive by qPCR (see Table 12c). These samples may reflect isolates that were not producing toxin and so not causing disease at the time of the diarrhoea, i.e. may have been carried toxigenic strains (colonisation). The primer targets the tcdB toxin gene, so would not have identified non-toxigenic strains. The reference laboratory test for C. difficile is an EIA-based test for toxin; however, as this test is known to report false positives and false negatives, the samples chosen for this investigation had been confirmed C. difficile positive by culture.

TABLE 12(c)

Sensitivity and specificity of MassCode assay compared with microbiology and microbiology plus PCR reference, and discrepancies: false positives and false negatives vs. standard reference

TABLE 12(a)

Sensitivity and specificity of MassCode assay compared with microbiology and microbiology plus PCR reference, and discrepancies: sensitivity

TABLE 12(b)

Sensitivity and specificity of MassCode assay compared with microbiology and microbiology plus PCR reference, and discrepancies: specificity

Higher rates of identification of C. difficile on moving from toxin-based tests to NAAT- or PCR-based tests have been widely reported, and identification of C. difficile on the basis of PCR alone is not recommended, as outcomes are similar in PCR-positive toxin-negative and PCR-negative toxin-negative individuals.⁵ All 13 samples testing postive for C. difficile in the laboratory but missed by MassCode were identified by qPCR, suggesting that these errors were simply due to the multiplex assay and the loss of sensitivity seen in large multiplex assays, rather than to the target primers or the extraction method.

For the other species, fewer of the MassCode-positive reference-negative samples were confirmed as containing the target organism using qPCR (50% for Campylobacter spp. and norovirus, 12% for S. enterica). S. enterica had a particularly high rate of false-positive calls by MassCode among the main target organisms, with only two of the unexpected positives being confirmed by qPCR. Most of those positive on PCR had evidence of a test negative on the microbiology system, i.e. not isolated rather than not tested for in error. Although this could, in theory, represent detection of lower levels of organisms than required to cause disease by MassCode, the relatively high limits of detection for the MassCode assay for S. enterica suggest that it is more likely these are genuinely false-positive calls by the multiplex assay.

For Campylobacter spp., most (17/19) of the reference laboratory-positive MassCode-negative samples were positive on qPCR, again, suggesting that these errors were simply because of the multiplex assay, not the target primers or the extraction method. Just over half (19/34) the S. enterica laboratory-positive, MassCode-negative samples were positive using a single PCR, suggesting that both primers or extraction method and multiplexing the assays were playing important roles in the lower sensitivity for this organism. Lower PCR positivity for norovirus may reflect degradation of the single-stranded RNA even with a single assay or possibly sample degradation with storage.

Interestingly, of the 181 samples positive for Campylobacter spp. using both reference laboratory and MassCode assays, MassCode identified 130 C. jejuni infections, 10 C. coli infections and 41 co-infections with both C. jejuni and C. coli. It is not known whether these represent genuine co-infections or cross-reactive primer pairs, as Campylobacter is not routinely speciated in the service laboratory.

Discrepant results for additional organisms

Unexpected positive samples were all retested by the appropriate PCR assay, as for the main organisms. E. coli, S. Typhi, Shigella spp., Cryptosporidium spp., and Giardia spp. all also showed a high rate of false-positive calling (Table 14). However, some unexpected positives were also confirmed by PCR, including four (0.4% of all samples) E. coli O157 positives (three of which had definitely had faecal culture without identifying this pathogen), 13 (1.4%) Giardia spp. (none tested for parasites in the service microbiology laboratory), seven (0.7%) Cryptosporidium (one sample tested for this and no oocysts seen) and three (0.3%) Shigella spp. (two had faecal culture without identifying this pathogen). Nevertheless, these unexpected true positives were identified at the cost of a substantial number of false positives, particularly for rarer species, Cryptosporidium spp. and Giardia spp., with 7% and 13%, respectively, of the total 948 samples testing positive for these species by MassCode.

Co-infections

Co-infections were defined by either expected positives from reference standard testing or unexpected positives by MassCode followed by qPCR to confirm the unexpected positive. C. jejuni and C. coli were included as Campylobacter spp. for this analysis. Of the 948 samples, 32 (3.4%) were confirmed positive for two of the four main organisms (Table 15). Overall, 46 (4.9%) of the 948 samples were confirmed positive for more than one organism, only two of which were confirmed to be positive for more than two organisms. Co-infections with parasites (Cryptosporidium spp. and Giardia spp.) accounted for 11 of the co-infections, whereas co-infections with C. difficile and Campylobacter spp. or norovirus were most common. Of note, only eight Cryptosporidium-positive samples were found in total (one known co-infection, seven unexpected positives), but six out of eight organisms were present with another pathogen, suggesting this organism might be more commonly carried than previously suspected. All of the four unexpected E. coli O157 organisms found were also found with other, more common species, as were five of the unexpected 13 Giardia spp. organisms.

TABLE 15

Occurrence of co-infections

As a consequence of the relatively low sensitivity and specificity of the MassCode assay, far more samples were identified as having co-infections with enteric pathogens using these results alone (Figure 11). A total of 48 (5.1%) were MassCode positive for two of the four main organisms; eight samples were microbiologically negative. Of the 948 tested samples, only 241 (25%) had no organism of any type identified by MassCode, 181/295 (61%) of those microbiologically negative compared with 60/653 (9%) positive for any of the main four pathogens. In one sample up to six organisms were identified by MassCode; 159 samples contained two organisms, 55 samples contained three organisms, 17 contained four organisms and four samples contained five organisms. As expected from the results above, Giardia spp. and Cryptosporidium spp. were commonly incorrectly identified as additional co-infecting organisms.

FIGURE 11

Number of pathogens in samples by MassCode assay. (a) Including only main pathogens (Campylobacter spp., C. difficile, norovirus and S. enterica); and (b) including all 11 pathogens.

Sequencing Cryptosporidium product

The sections Specificity of MassCode for additional organisms and Discrepant results for additional organisms demonstrated that, although Cryptosporidium spp. were detected in high numbers by MassCode, 94% of these were confirmed to be false positive by singleplex PCR. Results from the sequencing of the Cryptosporidium product confirmed that the primers targeting Cryptosporidium SSU rRNA were also amplifying Candida spp. Of the four clinical samples tested, two that were called positive for Cryptosporidium spp. by MassCode were found to have produced Candida spp. products rather than Cryptosporidium products. This result helps to explain the high false-positive call rate by MassCode, as Candida spp. could be distinguished by melt curve analysis when tested by singleplex SYBR Green PCR. Although not confirmed through sequencing, it is possible that the high false-positive rate seen in Giardia (only 10% of MassCode positives were confirmed positive by singleplex PCR) may also be as a result of unspecific primer pairs being used, as the Giardia primers selected for use in MassCode also target SSU rRNA.