Survey development

The surveys (survey A is in Appendix 1 and survey B is in Appendix 2) were developed in multiple stages. They were initially drafted in web form and revised by Antaine Stiobhairt, Alasdair MJ MacLullich and Susan D Shenkin based on previous literature and personal clinical experience of delirium. The revised surveys were reviewed by members of the study team, who assessed face validity, structure and clarity, and further revisions were made. Subsequently, survey A was piloted with 19 additional health-care practitioners outside the study and survey B was piloted with five. In each case, participants were asked to comment on content and any technical problems. Minor amendments were made to both surveys.

In both surveys, the items were presented in a fixed order (with no randomisation) and included multiple choice, ranking, five-point Likert scale and open-comment response formats. Response options including ‘N/A’ (not applicable), ‘don’t know’ and ‘other (please specify)’ were provided throughout the survey, and the majority of questions were optional in order to minimise response coercion and attrition. The surveys began with six demographic items, two of which (career stage and specialty) were presented only to respondents who had a primary qualification in medicine. Some additional questions were presented to those with this qualification (see Appendix 1). Both surveys finished with an open-comment box in which respondents were invited to ‘comment on any of the issues raised in this survey or additional issues surrounding the detection and assessment of delirium that have not been addressed’. No incentives were offered for survey completion.

The surveys were considered to be service evaluations, as the participants were anonymous health-care practitioners and the surveys were of current practice; thus, formal research ethics approval was not required. The surveys were hosted on the internet using Survey Monkey (www.surveymonkey.com). Invited recipients who clicked on the hyperlinked URL in the e-mail were initially presented with a web page that described the study in greater detail and explained that participation was anonymous, that no computer location information or cookies would be collected, and that the results would be published and may involve direct quotations. Potential respondents were then presented with the question ‘Do you agree to participate in this survey and consent to the potential use of your anonymised responses as described above?’. Those who chose ‘I agree’ proceeded to the survey, whereas those who chose ‘I do not agree’ were directed to a page that explained that they had to agree if they wanted to participate. As it was anonymised, the survey did not allow for checking for multiple responses from single users. No time-stamp data for individuals were analysed.

Participants

For Survey A, the health-care practitioners of interest were medical practitioners, nurses, occupational therapists and physiotherapists working in the UK who came into contact with delirium as part of their daily routine. These professional groups were selected as they were numerically large and came into frequent contact with patients with delirium. A list of e-mail addresses of potential respondents was generated through networks of study authors and through internet searches focused on, but not limited to, EDs, ICUs, acute assessment units/medical assessment units (MAUs), elderly care, orthopaedics, oncology, stroke units and palliative care, as delirium is common in these settings. Contact details of relevant associations/societies and trusts/health boards were also obtained through internet search engines.

A standard invitation e-mail was sent to individual practitioners on 30 September 2014. Amended versions of this e-mail were sent to trusts/health boards across the UK, and to relevant societies/associations (geriatrics, psychiatry, acute medicine, nursing, palliative care, etc.). All recipients were asked to forward the e-mail to relevant practitioners in their own contacts list and to consider displaying an A4 poster highlighting the survey in their staff areas. Reminder e-mails were sent on 21 October 2014 and 18 November 2014. Each of the study collaborators forwarded the e-mail to their contacts informally during this time. The survey was closed on 3 January 2015, by which time participation had tapered off.

Survey B was intended for any health-care practitioners worldwide who had used the 4AT to screen patients for delirium. A convenience sample of suitable practitioners was obtained through multiple methods. A standard invitation e-mail was distributed to practitioners working in units where the 4AT was known to be in use on 4 November 2014. Practitioners who completed survey A (Delirium Detection and Assessment in Clinical Practice), who had confirmed that they had used the 4AT and who had provided their e-mail address were e-mailed on 6 November 2014. All recipients were asked to forward the e-mail to 4AT users in their own contacts list and to consider displaying an A4 poster highlighting the survey in their staff areas. Reminder e-mails were sent on 2 December 2014. Each of the study collaborators forwarded the e-mail to their contacts informally during this time. Posters were also displayed in staff areas of units in the Royal Infirmary of Edinburgh that use the 4AT, and paper slips containing the link were handed to practitioners informally. The survey was closed on 22 January 2015, by which time participation had tapered off.

Data analysis

Quantitative data were analysed using R version 3.0.2 (2013; R Core Team, The R Foundation for Statistical Computing, Vienna, Austria). The majority of the data had non-normal distributions and heterogeneous variance across groups; therefore, medians and interquartile ranges (IQRs) are reported throughout and non-parametric statistical tests were used.

For comparative data in survey A, the threshold for statistical significance was a p-value of < 0.05. Between-group analyses were carried out using chi-squared tests where both variables were categorical, using Kruskal–Wallis tests with Holm–Bonferroni method followed by Mann–Whitney U-tests. For items whose responses were given on five-point Likert scales, verbal responses were converted to the corresponding numeric responses 1–5 to facilitate between-group analyses, and analysed as continuous variables. Mean scores and standard deviations (SDs) were reported for these converted responses, as the median and IQR rarely varied from 3 and 2–4, respectively, owing to the restricted scoring range of 1–5. Between-group analyses involving work settings were carried out on participants who worked in single inpatient settings only, in order to avoid the possible confounding effects of working in multiple settings. Owing to the large number of significant pairwise comparisons, details of significant effects were not reported in text when the effect size r < 0.15, but were included in tables only. Unanswered items and sample attrition resulted in missing data and fluctuating sample sizes across and within items; therefore, the results are based on the total number of responses for each item, excluding cases of missing data and when respondents chose ‘N/A’ or ‘don’t know’, unless stated otherwise. No statistical corrections were applied.

Qualitative data submitted through open-comment fields were reviewed informally and the decision on which quotations to report was based on what was judged to contribute to improve the implementation or refinement of the 4AT.

Survey A

A total of 2671 practitioners agreed to participate in the survey. Of those, 172 stopped before completing the core demographic questions; 137 completed these questions but did not continue; 12 worked outside the UK; 41 had a primary professional qualification in an area other than medicine, nursing, occupational therapy or physiotherapy; for two their primary professional qualification could not be verified; and one was retired. These cases were excluded. Data from 2306 (86%) respondents were retained. A summary of the respondent characteristics is shown in Table 1.

TABLE 1

Survey A: respondent characteristics

All estimates relate to the frequency of respondents except where stated otherwise. Numbers in cells pertaining to settings are not equal to the total sample size, as respondents could choose multiple settings.

Detection of delirium

When respondents were asked their opinion on what percentage of patients with delirium in their unit had their delirium diagnosed and documented, using the bandings ‘0–20%’, ‘21–40%’, ‘41–60%’ ‘61–80%’ and ‘81–100%’, the responses for each banding were 14% (275/2010), 18% (352/2010), 22% (449/2010), 26% (526/2010) and 20% (408/2010), respectively. This suggests that those surveyed feel that there is substantial underdiagnosis of delirium, with a minority of units perceived to be achieving at least 80% (estimated) diagnosis rates. Estimates of diagnosis rates differed significantly by clinical area, with EDs showing the lowest estimates (mean 2.6, SD 1.12) and acute inpatient medical and rehabilitation settings showing higher estimates (mean 3.45, SD 1.22, and mean 3.6, SD 1.3, respectively). Overall, respondents believed that the largest contributor to the underdiagnosis of delirium in their units was ‘difficulty discriminating delirium from dementia’, with 44% (907/2072) stating that this made a ‘large’ or ‘very large’ contribution. This was followed by ‘lack of staff confidence in assessment’ (39%, 809/2050) and ‘lack of staff knowledge of delirium’ (39%, 817/2083). Notably, the difference between all of these contributors is limited to a range of 11%, and more than one-third of respondents considered each of the listed contributors to have a ‘large’ or ‘very large’ effect, indicating that respondents consider a complex combination of factors to be responsible for the underdiagnosis of delirium.

Regarding the role of different disciplines in ‘flagging potential cases’, ‘screening high risk patients’ and ‘making a formal diagnosis’, responses indicated that virtually the whole sample considered doctors to be responsible for diagnosis, with other disciplines considered to have a much smaller role in this. Nurses were considered to have the most important role in screening high-risk patients, whereas physiotherapists and occupational therapists were considered to have the main responsibility for ‘flagging potential cases’ (Table 2).

TABLE 2

Survey A: respondents’ answers when asked to indicate which of the three tasks different professional groups should consider their own duty with regard to delirium detection

Survey B

A total of 117 practitioners agreed to participate in the survey. Fourteen stopped before completing the core demographic questions (items 1–4) and three stated that they had never used the 4AT in clinical practice. These cases were excluded, giving a final sample of 100 (88%) respondents. The geographical distribution was Scotland (n = 62), England (n = 28), Italy (n = 5), the USA (n = 3) and Australia (n = 2). Appendix 3 gives a summary of respondent characteristics.

In addition to the 4AT, a large proportion of respondents reported that they had used the CAM (68%, 61/90). Among the 28 respondents who named additional tools they had used, the Abbreviated Mental Test (n = 15) and the MMSE (n = 8) were the two most commonly mentioned (see Appendix 4).

All respondents reported having at least moderate confidence in their ability to detect delirium (moderate 28%, 28/100; high 52%, 52/100; and very high 20%, 20/100). The interval from which the 4AT was first used by the individual to the time of completing the survey varied across the sample [‘< 1 month’, 10% (9/90); ‘1–6 months’, 21% (19/90); ‘7–12 months’, 18% (16/90); and ‘> 1 year’, 51% (46/90)].

A detailed summary of user opinions on the 4AT, collected as part of survey B, is provided in Appendix 5. Regarding general opinions on the 4AT, 84% (59/70) had positive views, 14% (10/70) had neutral or mixed views and one respondent (1/70) had a negative view. When respondents were asked how often they used the 4AT with patients at risk of delirium, 33% (30/90) stated ‘never/rarely’ or ‘sometimes’, 14% (13/90) said ‘about half of the time’ and 52% said ‘frequently’ or ‘almost always/always’. When respondents were asked if they thought that using the 4AT as part of routine assessment was feasible in their unit, most (95%, 81/85) said ‘yes’, with many referring to the ease (39%, 15/38) and speed (29%, 11/38) of use. One question addressed the extent of knowledge of delirium that respondents thought necessary for health-care practitioners to have in order to use the 4AT effectively: 7% (6/84) stated ‘none/very little’, 58% (49/84) stated ‘some’ or ‘a moderate amount’ and 35% (29/84) stated ‘quite a bit’ or ‘an extensive amount’. With respect to specific training in use of the 4AT for health-care practitioners to be able use the tool effectively, 17% (14/84) stated ‘none/very little’, 60% (50/84) stated ‘some’ or ‘a moderate amount’ and 24% (20/84) stated ‘quite a bit’ or ‘an extensive amount’.

When respondents were asked whether or not the 4AT was used as part of routine assessment by them or others in their unit, 69% (57/83) said ‘yes’, with two respondents commenting that they were currently carrying out audits of 4AT use in their units. Regarding barriers to using the 4AT in respondents’ units, several were identified (Table 3). The use of an alternative tool was not considered by most to be a significant barrier. Opinions on other barriers showed a broad distribution across the sample, suggesting that implementation of delirium assessment tools is complex. Two particular comments from the free-text comments emphasised that the extent of 4AT use depends on broad systemic and cultural factors and not simply on the merits of its utility:

TABLE 3

Survey B: the extent to which respondents thought that various barriers were preventing the 4AT from being used more regularly in their units

We tried it but then moved back to AMT + CAM as AMT already performed by Nursing staff as part of basic assessment.

Consultant, ED/MAU

We have done some improvement work with it but there was some resistance from colleagues about using it as a screening tool.

Speciality trainee level 3 or above (ST3+), MAU/Medicine of the elderly (MoE)

To gather information about ease of use of the 4AT, respondents were presented with descriptions of three patient groups and asked to indicate their typical experience of using the 4AT with each group (see Appendix 2). For ‘drowsy patients who cannot produce verbal responses’, 51% (36/70) reported that using the 4AT was ‘very easy’ or ‘easy’, 17% (12/70) reported that this was ‘neither easy nor difficult’ and 31% (22/70) reported that this was ‘difficult’ or ‘very difficult’. For ‘patients with dementia who are alert and able to converse’, 77% (60/78) reported that this was ‘very easy’ or ‘easy’, 17% (13/78) reported that this was ‘neither easy nor difficult’ and 6% (5/78) reported that this was ‘difficult’ or ‘very difficult’. For ‘patients who are agitated and distressed’, 44% (33/75) reported that this was ‘very easy’ or ‘easy’, 20% (15/75) reported that this was ‘neither easy nor difficult’ and 36% (27/75) reported that this was ‘difficult’ or ‘very difficult’.

Respondents were also asked how long items 1–3 (alertness, abbreviated mental test-4, attention) of the 4AT typically take to complete: 12% (10/81) stated ‘< 1 minute’, 54% (44/81) stated ‘1–2 minutes’ and 33% (27/81) stated ‘3+ minutes’. Some respondents added that the time taken is ‘affected by deafness’, that they usually have a ‘conversation at the same time’ and that ‘longer time usually hints at worse performance’. Most of those who commented explained that the time taken to obtain collateral history from patients’ family, carers, GP or medical records was highly variable. These findings confirm that the bedside components are brief in most patients but that item 4 (determination of acute onset) can be time-consuming.

A detailed account of responses to survey B questions along with examples of free-text responses is reported in Appendix 3. In addition to giving feedback on the 4AT itself, respondents were asked whether or not they would suggest making any changes to it. The majority did not propose any changes, with 6–10 comments on each item and the guidance notes. Respondents’ comments could be grouped into the following themes: changes to item content, clarity, visual presentation, the scoring system and general comments. Examples are shown in Box 1. There were also some queries about validity and diagnostic accuracy of the individual items. Similarly, a small number of respondents suggested including ‘time’ alongside the four items of the AMT4.

BOX 1

Examples of suggested changes to the 4AT from free-text comments in survey B

The findings of survey B were discussed with members of the study team, and considered in the light of other information, including additional external validation studies that had been published, the clinical service evaluations that had been collected as part of the study process, and other feedback about the 4AT from outside the study. The team decided that, in the absence of consistent feedback concerning a potential specific change to the 4AT, the study process had determined that no change to the 4AT was required in advance of the diagnostic accuracy study.

Survey A

Survey A had a substantial number of respondents and is, to our knowledge, the largest survey on delirium to date. Because of the nature of the recruitment process, respondents were likely to have had an interest in delirium and/or to have been interested in the management of cognitive impairment more generally. Nevertheless, given the sample size, the results do suggest that there is increasing awareness of delirium among hospital staff in the UK. Additionally, respondents indicated that the majority of units in which they worked had guidelines for delirium detection. These findings indicate potentially positive trends in improving delirium detection and, therefore, care in the UK. However, other findings suggest that substantial challenges remain.

Nine out of 10 of respondents agreed or strongly agreed that formal diagnosis of delirium is important, that distinguishing between delirium and dementia is important, and that delirium treatment improves patient care. Although these data are from a potentially biased sample of health-care professionals, they support the notion that a substantial proportion of practitioners believe that delirium is worth diagnosing. This view in the context of the well-documented poor rates of delirium detection demonstrates a challenging paradox in mainstream clinical practice.

Some of the findings of this study help to address the question of why this paradox exists. One of the striking results is the range of terminology used to describe a patient with ‘recent onset drowsiness’ who is ‘not producing responses but is responding intermittently to one stage commands’. This clinical scenario is clearly consistent with a diagnosis of delirium according to DSM-5 criteria and the accompanying guidance notes, especially considering that delirium is much more common than alternative diagnoses. Yet only one-third of respondents favoured the formal term ‘delirium’ being applied to this case, with several non-diagnostic, ill-defined terms, such as ‘obtundation’, collectively being suggested more often. Notably, a follow-up question asked respondents if the patient described at the beginning of this paragraph had delirium, to which two-thirds of respondents responded that this was ‘likely’ or ‘very likely’. Additionally, a different question directed at medical practitioners found that 72% believed that the patient needed to be able to produce verbal responses to allow bedside assessment of delirium; this view is not aligned with guidance in the DSM-5, which deems it possible to assess delirium in non-comatose patients with acute mental status disturbance who are incapable of speech. These findings indicate that there is remarkable inconsistency in the terms used to describe the clinical states most compatible with an initial diagnosis of delirium. The results in relation to the variable terminology used in the organisations of respondents parallel the findings from individual practitioners. The consequences of using inconsistent terminology are potentially serious, including failure to apply agreed treatment pathways, impaired communication among staff and an absence of adequate communication of the diagnosis to patients and carers. Moreover, an incoherent approach to diagnosis among senior staff makes attempts to provide effective training to junior practitioners and students much more challenging. Thus, these findings have broad implications for education and training, including continuing professional development.

Practitioners showed a range of levels of confidence in their own ability to detect delirium. Given that the sample is likely to be biased towards those with an interest in the condition, it is of concern that at least a substantial minority of respondents, including medical practitioners, have very low to moderate confidence. Given that delirium is very common, affecting at least one in eight hospital patients, this lack of confidence does not stem from unfamiliarity with the condition; rather, it is likely to result from insufficient education and training. The general incoherence around approaches to delirium detection may also contribute to the lack of confidence in many practitioners. The findings showed that just over half of respondents had ever used a specific delirium detection tool; non-specific cognitive tests were more widely used. Related to this, although around two-thirds of units had delirium guidelines, respondents reported that in only a minority of these were the guidelines followed.

Another informative finding concerns the perceptions of the responsibilities of different disciplines with respect to delirium. In particular, respondents mostly felt that nurses were responsible for screening for delirium but not for diagnosing it. It is not rational or pragmatic to restrict diagnosis of such a common condition to medical staff alone, when the diagnosis can usually be made readily with bedside assessment and informant history (or personal knowledge of the patient). On the contrary, it can be argued that nurses who have appropriate training (including an awareness of the DSM-5 criteria) are in a strong position to make a diagnosis, because they have the most direct contact with the patient of any health-care practitioner. Notably, the prompt diagnosis of delirium is advocated by many policy-makers, such as Healthcare Improvement Scotland. This finding has implications for both policy and clear decision-making around the explicit roles of doctors and nurses in detecting delirium and initiating early care. With respect to physiotherapists and occupational therapists, an awareness of the main features of delirium, perhaps coupled with the use of a screening tool such as the 4AT, could lead them to report potential delirium to colleagues qualified to make a diagnosis.

Survey B

Survey B was directed at respondents who had experience of using the 4AT. The final sample size was 100 and comprised mostly doctors and nurses. A large majority of respondents reported carrying out delirium assessments ‘frequently’ or ‘almost always/always’, meaning that this is a sample that is likely to include a substantial proportion of professionals experienced in delirium assessment. In addition to responses to the structured survey questions, multiple free-text comments were provided.

Taking the findings as a whole, respondents generally viewed the 4AT as a useful, rapid and practical tool. Several comments were made about potential changes. However, no clear problem with the 4AT emerged consistently. Many of the issues raised relate not to the 4AT specifically but to general challenges in the assessment of delirium, such as the availability of informant history and the time it can sometimes take to get this history, and basic knowledge of the features of the syndrome itself. Other issues relate to possible modifications of the cognitive tests used, the time frame over which altered mental status is considered to indicate delirium, and so on. Many of these suggestions are reasonable and reflective of variations in accepted, mainstream practice. However, given that the current version of the 4AT is in wide use, supported by several validation studies and mentioned in several policy statements and guidances (see Chapter 3), in the absence of strong positive evidence in favour of these modifications being introduced, retaining the current version is the most pragmatic option.

Around one-third of practitioners reported that it was ‘difficult or ‘very difficult’ to complete the 4AT for ‘drowsy patients who cannot produce verbal responses’. Items 2 and 3 of the 4AT allow for scoring such ‘untestable’ patients, and item 1 allows for scoring drowsy patients. This finding suggests that education about the features of delirium with respect to reduced arousal would be helpful, and specific training on the 4AT concerning this issue would reduce the uncertainty reported by some practitioners. With respect to training, most respondents stated that at least some training in using the 4AT would be needed for the tool to be used effectively. The proportions of respondents stating this were similar to the proportions responding to the question about whether or not training in delirium in general would be required to use the tool. Although the 4AT was designed to be simple and practical, and usable without specific training, the survey findings suggest that users must have a basic understanding of delirium if they are to use the 4AT. This should include understanding that a reduced level of arousal is commonly seen in delirium and contributes to that diagnosis. A number of comments suggested that more guidance is needed about what to ask carers or contacts in terms of changes in patient behaviour. This knowledge is best classed as part of a practitioner’s general information about delirium detection; nevertheless, given the lack of other approaches to delirium detection in routine care, more information on how this should be done in the context of using the 4AT would be valuable. Additionally, in conjunction with this, some training in the use of the 4AT would appear to be beneficial in promoting its effective use. Given the lack of space on the one-page 4AT form, such additional education and training on delirium in general and on the 4AT would best be provided on easily accessible websites, including www.the4at.com.

Some queries arose about the scoring of the cognitive test items. Cognitive tests used in hospital practice show variable diagnostic accuracy for dementia and delirium,^46,52,95,96 and no single test provides sufficiently good performance for it to be used in all contexts. In addition, cognitive tests performed in isolation inform clinical judgement, but cannot be performed as diagnostic tests alone. Therefore, practitioners using the AMT4 and Months Backwards tests need to be aware of both the value and the limitations of the information provided by the test results. For practitioners seeking further information about the cognitive test items in the 4AT, the URL of the 4AT website is given on the form (and, indeed, is easily found through an internet search), and an up-to-date list of specific 4AT validation studies, as well as relevant studies relating to the cognitive test items, is provided on that website.

A need for both the scoring and the guidance to be on the paper or electronic documentation was also identified. This sometimes seems to have been lost when the 4AT has been incorporated into a larger assessment such as an electronic patient assessment form. This could be addressed by providing clear links to the 4AT guidance notes or the website.

The current findings emphasise that the use of the 4AT is not based solely on the utility of the tool itself, but also depends on broader systemic and cultural factors. All of those participating in survey B had some experience of delirium assessment, and yet there was some evidence of lack of knowledge about some general aspects of delirium, which had an adverse effect on participants’ understanding of how to use the 4AT. Furthermore, respondents identified external factors that hindered their use of the 4AT, and presumably their use of any other delirium assessment tool.