Registered participants

A total of 155 health-care professionals (72 ophthalmologists and 83 optometrists) registered their interest in the ECHoES trial. Of these, 62 ophthalmologists and 67 optometrists consented to take part. Everyone who registered an interest was eligible for the trial; those who did not consent either did not return their consent forms or were no longer required for the trial. See Figure 3 for details.

FIGURE 3

Flow of participants.

Recruitment

Participants were initially recruited between 1 June and 9 October 2013. However, as participants progressed through the training stages, it became apparent that the withdrawal rate of optometrists was higher than expected and that more would be required if the planned sample size were to be met. Recruitment was therefore reopened between 13 February and 6 March 2014, when a further seven optometrists consented to take part. A number of participants withdrew or were withdrawn by co-ordinating staff throughout the trial, mostly because they missed webinars or failed their assessment of training vignettes (see Figure 3 and Withdrawals for details). The final participant completed the main study vignettes on 21 April 2014. As planned, 48 ophthalmologists and 48 optometrists completed the full trial.

At the start of the trial we were unsure how many participants we would need to recruit in order to meet our target of 48 participants in each group. Therefore, we over-recruited at the consent stage and asked a number of participants to complete the webinar and then ‘wait and see’ whether or not we needed them to participate in the main trial. We also slightly over-recruited at each stage of the trial to account for dropouts. This resulted in a small number of participants being withdrawn at various stages of the trial because they were no longer required.

Withdrawals

During the ECHoES trial, withdrawals could occur for a number of reasons. First, participants could withdraw or be withdrawn between consenting and receiving their training vignettes if the mandatory webinar training was not completed, they no longer wanted to take part or they were no longer needed for the trial. This occurred for six ophthalmologists and six optometrists (see Figure 3 for details). Second, participants could be withdrawn if the threshold performance score for the assessments of their training vignettes was not attained. Of the 54 ophthalmologists who completed their vignette training, 48 (88.9%) passed first time, two (3.7%) passed on their second attempt and four (7.4%) failed their second attempt so were withdrawn. Of the 57 optometrists who completed their vignette training, 38 (66.7%) passed first time, 11 (19.3%) passed on their second attempt and eight (14.0%) failed their second attempt and were withdrawn from the trial. Two ophthalmologists and one optometrist who passed their training vignettes were withdrawn from the trial as the target sample size (48 in each group completing main assessments) had already been reached.

Numbers analysed

Ninety-six participants, 48 ophthalmologists and 48 optometrists, were included in the analysis population. For the primary and secondary outcomes, all participants were included as by design there were no missing data.

Reference standard classifications

The reference standard classified 142 (49.3%) of the 288 vignettes as reactivated, 5 (1.7%) as suspicious and 141 (49.0%) as quiescent.

Participant characteristics

The characteristics of participants collected for the trial included only age, gender and date of qualification for the participant’s profession. As participants were not randomised to the two comparison groups, baseline characteristics were presented descriptively and formally compared. Table 5 shows that the gender balance and average ages were similar among optometrists and ophthalmologists [mean 43.1 years (SD 10.1 years) and 42.2 years (SD 8.0 years), respectively; 50.0% vs. 43.8% women]. Optometrists had on average significantly more years of qualified experience than ophthalmologists [median 17.4 years (IQR 10.1–28.4 years) and 11.4 years (IQR 4.8–16.9 years), respectively].

TABLE 5

Characteristics of participants

Primary outcome

The primary outcome (correct lesion classification by a participant compared with the reference standard: 2.5.1) was achieved by the ophthalmologists for 1722 out of 2016 (85.4%) vignettes and by the optometrists for 1702 out of 2016 (84.4%) vignettes (Table 6). The odds of an optometrist being correct were not statistically different from the odds of an ophthalmologist being correct (OR 0.91, 95% CI 0.66 to 1.25; p = 0.543). The ability of optometrists to assess vignettes is non-inferior (both clinically and statistically) to the ability of ophthalmologists, according to the prespecified limit of 10% absolute difference (0.298 on the odds scale; illustrated by the dashed black line on Figure 4). In this primary outcome model, the variance attributed to the participant random effect was far smaller than that of the vignette random effect (0.360 and 2.062, respectively).

TABLE 6

Correct lesion classification (primary outcome), sensitivity and specificity

FIGURE 4

Comparison between optometrists and ophthalmologists for the primary outcome. Black dashed line represents the non-inferiority limit of 0.298. Effect size point estimate is shown as the OR and the error bar represents the 95% CI. The solid vertical line (more...)

The median number of correct lesion classifications (compared with the reference standard) by individual participants was 37 (IQR 35.0–38.5) in the ophthalmologist group compared with 36 (IQR 33.0–38.0) in the optometrist group. The lowest number of correct lesion classifications was 26 out of 42 in the ophthalmologist group and 24 out of 42 in the optometrist group; the highest number of correct lesion reactivation decisions was 41 out of 42, achieved by one ophthalmologist and three optometrists. It was recommended by a reviewer that the relationship between years of experience and the number of correct responses of ophthalmologists should be explored. Appendix 4, Figure 27 suggested no clear relationship. Appendix 4, Figure 28 shows a plot of individual optometrist participant scores against their paired ophthalmologist counterparts for the 48 different sets of vignettes. The sensitivity and specificity (see Table 6; described in Chapter 2, Group comparisons) and detailed breakdown of the participants’ classifications (Table 7 and Figure 5) show the agreement between participants and the reference standard in more detail. Figure 5 shows that optometrists were more likely than ophthalmologists to correctly classify a vignette as reactivated (80% vs. 74.0%), but were less likely to correctly classify a vignette as quiescent (65.7% vs. 81.7%).

TABLE 7

Participants’ lesion classifications compared with the reference standard

FIGURE 5

Participants’ lesion classifications for vignettes classified as reactivated or quiescent by the reference standard. The 35 observations in each group for the five vignettes classified by the reference standard as suspicious are not shown in this (more...)

A post-hoc analysis to look at this subgroup effect was undertaken. The interaction between participant group and reference standard vignette classification (reactivated vs. quiescent/suspicious) was statistically significant (interaction p < 0.001); the odds of an optometrist being correct were about 50% higher than those of an ophthalmologist if the reference standard classification was reactivated (OR 1.52, 95% CI 1.08 to 2.15; p = 0.018) but about 70% lower if the reference standard classification was quiescent/suspicious (OR 0.27, 95% CI 0.17 to 0.44; p < 0.001).

Secondary outcomes

Serious sight-threatening errors

Serious sight-threatening errors could occur only for the vignettes which were classified as ‘reactivated’ by the reference standard. These errors occurred in 62 out of 994 (6.2%) of ophthalmologists’ classifications and 57 out of 994 (5.7%) of optometrists’ classifications; this difference was not statistically significant (OR 0.93, 95% CI 0.55 to 1.57; p = 0.789).

Each participant viewed between 15 and 27 ‘reactivated’ vignettes and the most sight-threatening errors made by a single participant was eight (out of 25) in the ophthalmologists group and five (out of 19) in the optometrists group (Table 8). Table 8 also shows the number of non-sight-threatening errors, that is, participants classifying vignettes as reactivated when the reference standard was quiescent. This type of error occurred more frequently in the optometrists group (105 out of 987; 10.6%) than in the ophthalmologists group (35 out of 987; 3.5%); this difference was not formally compared.

TABLE 8

Sight-threatening and non-sight-threatening serious errors

Lesion components

We did not attempt to achieve a consensus among the three experts for the individual features of the vignettes. Therefore, responses given by the professional groups for these individual features were formally compared with each other rather than with any reference standard. Optometrists judged lesion components to be present for all components except PED more often than ophthalmologists (Table 9 and Figure 6). This difference was particularly evident for DRT and exudates; the odds of identifying these components as present were more than three times higher in the optometrist group than in ophthalmologist group [OR 3.46, 95% CI 2.09 to 5.71 (p < 0.001), and OR 3.10, 95% CI 1.58 to 6.08 (p < 0.001), respectively]. SRF was also reported significantly more often by optometrists than ophthalmologists (OR 1.73, 95% CI 1.21 to 2.48; p = 0.002). The difference between the groups was of borderline statistical significance for blood (OR 1.56, 95% CI 1.00 to 2.44; p = 0.048). The differences between the groups for IRC and PED did not differ statistically [OR 1.00, 95% CI 0.61 to 1.65 (p = 0.985) and OR 0.91, 95% CI 0.47 to 1.79 (p = 0.786), respectively].

TABLE 9

Identification of lesion components by professional group

FIGURE 6

Lesion components. Point estimates for different lesion components are ORs and error bars are 95% CIs. The vertical dashed line illustrates line of no difference at 1.

Confidence ratings

The confidence ratings displayed in Table 10 show that ophthalmologists were clearly more confident in their decisions than optometrists. Ophthalmologists stated that they were very confident (5 on the rating scale) in their judgements for 1175 out of 2016 (58.3%) vignettes, whereas optometrists reporting the same level of confidence in their judgements for only 575 out of 2016 (28.5%) vignettes (OR 0.15, 95% CI 0.07 to 0.32; p < 0.001). For both groups, a confidence rating of 5 resulted in a correct answer over 90% of the time, but there did not appear to be any clear relationship between confidence and correctness for lower confident ratings, especially for optometrists (see Table 10).

TABLE 10

Confidence in classification of activity status of lesions by professional group

Key vignette information

It was stated in the protocol that the effect of key vignette features on correct reactivation decisions would be assessed. For interpretability we modelled the effect of these features on the number of incorrect classifications.

The influence of vignette features was investigated using Poisson regression, adjusting for the reference standard classification. Interactions between these vignette characteristics and professional group were tested but were not retained as they were not statistically significant at the 5% level. The outcome of this analysis can be seen in Figure 7: professional group, gender, cardiac history and age did not significantly influence the number of incorrect classifications. In contrast, the reference standard classification, smoking status and BCVA sum and difference were all statistically significant (p < 0.001, p = 0.005, p = 0.001 and p = 0.004, respectively). Vignettes of current smokers were more likely to be incorrectly classified than non-smokers, but differences between ex-smoker and non-smoker vignettes were not found [incidence rate ratio (IRR) 1.33, 95% CI 1.05 to 1.70, and IRR 0.91, 95% CI 0.76 to 1.10, respectively]. Vignettes with better BCVA (larger average BCVA over the two visits) were less likely to be incorrectly classified (IRR 0.996, 95% CI 0.993 to 0.998), while vignettes with a greater increase in BCVA from baseline to index visit were more likely to be incorrectly classified (IRR 1.017, 95% CI 1.005 to 1.028). Vignettes classified as reactivated by the reference standard were more likely to be incorrectly classified (IRR 3.16, 95% CI 2.62 to 3.81); this finding was in agreement with the raw data displayed in Figure 5 (noting that, in the two right-hand columns, classifications of quiescent and suspicious should be pooled for the total percentage correct).

FIGURE 7

Influence of key vignette information. Cardiac history included angina, myocardial infarction and/or heart failure. Point estimates show IRRs and error bars are 95% CIs. Line of no difference is illustrated by the vertical dashed line at 1. For items (more...)

Sensitivity analysis

For the primary outcome, a vignette classification was defined as ‘correct’ if both the reference standard and the participant classified the vignette as ‘reactivated’ or if both classified a vignette as ‘suspicious’/’quiescent’. A sensitivity analysis of the primary outcome was performed in which suspicious classifications were grouped with reactivated classifications instead of quiescent classifications. In this analysis, ophthalmologists correctly classified 1756 out of 2016 (87.1%) vignettes and optometrists correctly classified 1606 out of 2016 (79.7%). This difference was statistically significant (OR 0.51, 95% CI 0.38 to 0.67; p < 0.001), but the lower end of the CI did not cross the non-inferiority margin (0.298). Therefore, when correct classifications were redefined in this way, optometrists were statistically inferior but clinically non-inferior to ophthalmologists.

Additional (post-hoc) analyses

Vignette classifications compared with referral recommendations

It was of interest to see how lesion classification decisions related to referral decisions. These should be congruent (as referral decisions should be made based on classification decisions) but no such rules were imposed in this trial. Table 11 illustrates that, as expected, lesions classified as reactivated were usually paired with a recommendation for rapid referral to hospital (1679/1682, 99.9%) and lesions classified as quiescent were usually paired with a recommendation for review in 4 weeks (1526/1604, 95.1%). There was very little difference between the two professional groups in the extent to which lesion classifications and referral decisions were paired. Lesions classified as suspicious were often paired with a recommendation for review in 2 weeks (605/746, 81.1%), but rapid referral to hospital was not uncommon (131/746, 17.6%). Optometrists more often than ophthalmologists tended to pair suspicious classifications with review in 2 weeks (87.3% vs. 74.4%), whereas ophthalmologists tended to pair suspicious classifications with rapid referral to hospital more often than optometrists (24.8% vs. 10.9%).

TABLE 11

Referral recommendation by lesion classification decision

Duration of vignette assessment

The assessment durations were calculated for each participant as the difference between the time an assessment was saved and the time the previous assessment was saved. Therefore, this information was not available for all 4032 assessments because many participants took breaks between assessments (see Chapter 2, Missing data). The median durations of vignette assessment for all main study assessments were 2 minutes 21 seconds (IQR 1 minute 39 seconds to 3 minutes 27 seconds; n = 1835) for ophthalmologists and 3 minutes 2 seconds (IQR 2 minutes 5 seconds to 4 minutes 57 seconds; n = 1758) for optometrists. Assessment time, on average, reduced as experience increased, especially in the optometrists group; for ophthalmologists and optometrists respectively, the median assessment durations were 2 minutes 44 seconds (IQR 2 minutes 3 seconds to 4 minutes 40 seconds; n = 45) and 5 minutes 23 seconds (IQR 3 minutes 18 seconds to 10 minutes 3 seconds; n = 41) for the second main study vignette, and 1 minute 55 seconds (IQR 1 minute 19 seconds to 2 minutes 49 seconds; n = 48) and 2 minutes 31 seconds (IQR 1 minute 52 seconds to 4 minutes 26 seconds; n = 48) for the 42nd (final) main study vignette. Figure 8 shows this relationship.

FIGURE 8

Duration of vignette assessment. Point estimates show median assessment durations and error bars show IQRs.

Among vignette assessments for which duration was not missing, there were 1570 out of 1835 (85.6%) correct responses by ophthalmologists and 1492 out of 1758 (84.9%) correct responses by optometrists. Figure 9 illustrates the relationship between assessment duration and the percentage of correct responses; broadly speaking, shorter assessment durations were more likely to result in correct lesion assessments than longer assessment durations. This relationship was similar for both professional groups.

FIGURE 9

Correct responses by vignette assessment duration. Point estimates show the percentage of correct assessments and errors bars show exact 95% CIs.

Expert classifications for derivation of the reference standard

Each expert individually assessed the vignettes in order to develop the reference standard (see Chapter 2, Reference standard). The lesion classifications of the three experts were congruent for these assessments for 219 out of 288 (76.0%) vignettes (comprising 103 out of 219 classifications of reactivated and 116 out of 219 classifications of quiescent). The three experts then held a consensus meeting and jointly assessed the vignettes for which there was disagreement about the lesion classification. The presence or absence of lesion components in index images of these vignettes, and their change from baseline, were discussed in detail. Consensus lesion classifications were agreed which, together with the congruent classifications, made up the final reference standard. Experts did not attempt to reach consensus about specific lesion components. A few specific vignettes were reassessed by experts who felt that they had made errors about the lesion components in their original assessments. When carrying out these reassessments, data were collected only for assessments of the lesion components.

Table 12 shows the reference standard classification against each expert’s initial individual classification; 774 out of 864 (89.6%) of the individual classifications remained unchanged, with the rest (10.4%) being amended after group discussions. The responses that did not change are shaded green. The majority of the changed classifications involved an initial classification of suspicious by one expert.

TABLE 12

Reference standard compared with initial expert classification

Comparing lesion component classifications across experts

Although a reference standard was established for the overall lesion classification, no such standard was established for the lesion component classifications. Therefore, it was of interest to compare lesion component classifications across experts for the six lesion components. The experts could classify each lesion component as absent, present but not increased since baseline, or present and increased since baseline. The frequencies with which each expert identified different lesion components across all vignettes are shown in Table 13. Agreement between experts with respect to lesion components is shown in Table 14. Table 14 shows that agreement was best for blood and exudates; the three experts agreed about blood classification for 266 out of 288 (92.4%) vignettes and for exudate classification for 260 out of 288 (90.3%) vignettes. Table 13 shows that experts observed no blood or exudates on the about colour images of > 90% of the vignettes. Agreement for SRF and IRC was also high, with total agreement for 259 out of 288 (89.9%) and 243 out of 288 (84.4%) of the vignettes, respectively (see Table 14). Interestingly, for both these components, expert 2 disagreed with the other two experts slightly more often than experts 1 and 3 did with each other; Table 13 shows that expert 2 identified SRF to be present more often than experts 1 and 3, and identified IRC to be present less often than experts 1 and 3. Agreement between experts was lower for DRT and PED than for the other components, suggesting that classification of these components was potentially more difficult or less clear-cut. In addition, Table 14 shows the agreement between the experts about the overall lesion classification, prior to any discussions to agree the reference standard.

TABLE 13

Lesion component classifications of individual experts

TABLE 14

Comparison of lesion component classifications across experts

Participants’ views on the ECHoES trial training

This section includes information provided by all participants who completed the questionnaire on their opinions of the ECHoES trial training, regardless of whether or not they completed the study and were included in the analysis population. A total of 102 participants completed the questionnaire: 47 ophthalmologists (44 of whom passed training and became main study participants) and 55 optometrists (47 of whom passed training and became main study participants).

Overall, participants gave positive feedback about the training. In particular, both the ophthalmologists and the optometrists commented on the thorough nature of the webinars and how helpful the trial staff were, particularly the trial co-ordinator. Many optometrists were pleased to have taken part, finding the training to be ‘very useful’ and noting that the training had improved their confidence.

Well designed, a lot of thought and hard work to set and to run.

Optom262

Extremely useful and well structured.

Ophthalm131

I learnt a lot and am pleased I took the opportunity to take part.

Optom269

Table 15 shows that more ophthalmologists than optometrists found the web application harder to use. Of those who provided free-text responses, the majority (both ophthalmologists and optometrists) commented on the content of the training. Specifically, 10 participants felt that the quality of the OCT images was poor. Eight participants (five of whom were optometrists) also commented that it would have been helpful to have a side-by-side comparison of baseline and index images on the same screen, as it was difficult and time-consuming to have to change back and forth between images. Three participants commented that they used two different screens alongside each other to overcome this.

TABLE 15

Participants’ opinions on the ECHoES trial training

Some of the images were difficult to view as they were of poor quality.

Optom254

I found it time-consuming changing between the baseline and index scans. I speeded up once I solved this by having one set on my iPad^® and one set on my monitor.

Optom268

Only issue I had was that we were unable to compare the baseline and index images side by side which made assessing more difficult.

Optom225

Table 15 shows that 78% of optometrists felt that additional training may be required, compared with only 30% of ophthalmologists. One ophthalmologist and five optometrists elaborated on this, stating that they had found the content of the webinars to be confusing (particularly in terms of the questions about lesion components), and one ophthalmologist and three optometrists said they had found the training challenging.

Uncertainty about the some of the decisions I was making – not good for the nerves!

Optom256

Questions sometimes were confusing.

Ophthalm102

It was really challenging . . . you really had to think.

Ophthalm107

Years of ophthalmic experience did not appear to influence how well the training was received; ophthalmologist participants who felt that additional training was required had a median of 9.6 years of experience (IQR 4.6–15.3 years of experience), whereas ophthalmologist participants who felt training was completely sufficient had a median of 11.0 years of experience (IQR 5.6–16.1 years of experience). Almost all optometrists (96%) stated that they had revisited the webinar content, compared with half of ophthalmologists (49%; see Table 15). On average, the optometrists also spent over 1 hour longer revisiting the material than the ophthalmologists. Only 11% of ophthalmologists said that they had used other resources, compared with 67% of optometrists. Most participants used websites (25%) or a textbook (20%), had discussions with colleagues (6%) or looked through previous conference notes (3%). With respect to textbooks, the majority of participants had used Clinical Ophthalmology by Kanski and Bowling.²⁷