Background
The findings of WP1 (LIAISE) and the challenges of setting up WP3 (DOORWAYS) led to this add-on study. Organisational change in the NHS is often imposed via a top-down approach, which may not take into consideration the views of nursing staff. Indeed, change process issues such as poor involvement in planning, implementation and control of the project have been highlighted as potential barriers to success.51 The emotional response of staff to proposed changes may therefore be an important barrier to the success of implementation.
We propose a model in which negative staff perceptions of the daily pressures of working on an acute ward and negative staff perceptions of barriers to change lead to burnout and low job satisfaction, and this affects the quality of care delivered. In nursing studies, there has been limited focus on assessing how social, emotional and psychological barriers affect the implementation of new innovations. The relationship between psychological well-being and the acute ward setting in relation to planned changes may, in nursing, be distinct from other professions because nurses remain on the ward for the duration of their shift and are often expected to play a key role in delivering changes. There are several measures developed in health care that focus on the uptake of evidence-based practice in clinical areas by mental health providers and nurses.64–71 However, there are no measures that focus on general changes in a mental health setting.
What is needed in the literature on acute care is a psychometrically robust, brief, self-report measure reflecting staff experiences of delivering/innovating. A measure of this type would allow a clear assessment of changes to inpatient care following specific interventions to improve the environment and therapy provided. This add-on study was designed to generate such a measure.
Methods
Study design
This study was designed to develop and test this self report measure. The measure was generated through a process of stakeholder (staff) involvement.19–21 This approach directly involved nursing staff in order to ensure measures that captured an accurate picture of an acute care ward. The process of measure development used for VOCALISE (see Appendix 10 for full measure) was an adaption of that used for VOTE.
Mapping out the dimensions of inpatient care
In this initial phase, stakeholder groups were held to map out the dimensions of inpatient care as a topic guide. In developing VOCALISE, an interview schedule was refined from the literature and the focus group data as part of a consultation exercise between two nurses, one who was a senior nurse from the trust under study, an expert in qualitative methods/service user researcher and a clinical psychologist.
Measure generation
Sample
Nursing staff from all grades were asked to participate in the development of VOCALISE. The focus groups showed that staff were aware of the nursing hierarchy when discussing sensitive information. Therefore, individual interviews were held to collect data. Finally, two expert panels met to discuss the design of the new measure and to inform the ‘instructions for use’. Changes were made to the items and to the layout of the measure on the basis of this feedback. The draft measure was finally presented back to the original reference group for their comments.
Psychometric analysis
The final measure was intended as a self-report tool so studies of feasibility and acceptability were conducted to evaluate the burden of administering and completing the measure. In the feasibility study, 40 participants completed the measure including two additional questions assessing whether the measure was easy to complete and understand. Acceptability was assessed using two additional items collected from 115 participants. Test–retest reliability was assessed with participants who completed the measures on two occasions with an interval of 6–10 days. Internal consistency was assessed using Cronbach’s alpha.
Whether or not the new measures truly reflected the experiences of those providing care in the acute inpatient services (face validity) and covered the full spectrum of views (content validity) were explored as a result of the participatory methodology during the instrument development phase. We hypothesised that staff with positive perceptions of barriers to change would also have high levels of job satisfaction.199 This was tested using two groups: staff with positive perceptions of job satisfaction and staff with negative perceptions of job satisfaction.
Exploratory analyses
Do demographic characteristics affect perceptions of barriers to change?
A random-effects regression model was run to provide a clear picture of which demographic variables are true predictors of staff perceptions of barriers to change, controlling for all demographic variables, to test the following hypothesis derived from the literature: staff in managerial roles who (1) are from a younger age group and (2) are degree educated will have more positive perceptions of barriers to change.
Results
Mapping out the dimensions of inpatient care
As the concept of ‘perceptions of barriers to change’ is abstract, the reference group recommended that the interviewee be asked to consider a scenario in practice in which a significant change to clinical practice had occurred. A topic guide for interviews was created.
Measure generation
Sample
The sample involved in the development of VOTE included inpatient nursing staff from all grades within one London mental health trust. The catchment area of this trust is large and covers four inner and outer London boroughs. In total, 376 individuals were involved at the various stages ().
VOCALISE measure generation group
Interviews
Thematic analysis produced seven domains: ‘communication’, ‘generation of ideas’, ‘outcomes of changes’, ‘resistance’, ‘strategy’, ‘support and monitoring’ and ‘team dynamic’. Themes with fewer than 20 references were not included. Items were constructed around perceptions of the team/ward and perceptions of the self in relation to change.
Expert panels
Those who participated in the expert validation process confirmed that the content of the measure was relevant and that the content domain had been widely explored. There were no changes to the measure as a result of the final consultation.
Psychometric assessment
Feasibility and acceptability
The feasibility study (group one, n = 40) showed that VOCALISE was easy to complete (94% agreed) and easy to understand (100% agreed). Changes were made to the wording of some items. Those that seemed to be addressing more than one issue were simplified, while ensuring that these simplifications did not lead to the omission of any prevalent themes. Those items with poor or loaded phrasing were rephrased using more neutral language to give staff scope to agree or disagree. After these modifications all 23 items still remained.
The acceptability study showed that 73% thought the length of the questionnaire was about right. 20% enjoyed filling out the questionnaire, and 67.8% had neutral feelings. Finally, 107 (93%) did not find completing the items upsetting.
Total scores were calculated by totalling all items with no missing data. Negatively phrased items were reverse scored so that higher scores indicated a more negative perception of the ward. When comparisons to other measures were made, the same rule was applied to their scoring (i.e. high scores indicate poorer satisfaction).
Reliability
Test–retest reliability was assessed using data from group two (n = 42). Four items were unreliable with a kappa maximum < 0.39 and were dropped from the scale, leaving eight items with fair reliability (0.39–0.49), seven items with moderate reliability (0.50–0.56) and three items with substantial reliability (0.61–0.71).
Concordance between the total scores was good (total score, ρ = 0.76). However, a paired t-test showed that there was a significant difference between the two time points (t = –2.10; p = 0.04) (mean difference = –2, 95% CI –3.93 to –0.07). Test–retest reliability was therefore assessed according to staff group revealing that staff in direct care roles (n = 26) were likely to change their scores (t = –2.91; p = 0.008) (mean difference = –3.12, 95% CI –5.32 to –0.91). The scores of those in managerial roles were stable (t = 0.35; p = 0.73) (mean difference = 0.64, 95% CI –3.36 to 4.64; n = 11). The test–retest reliability of the ‘I’ statements and general statements was also assessed showing that over time, the general statements (t = –1.88; p = 0.07) (mean difference = –1.24, 95% CI –2.59 to 0.10; n = 37) were more likely to change than the ‘I’ statements (t = –1.50; p = 0.27) (mean difference = 0.14, 95% CI –1.82 to 0.27; n = 40).
The final Cronbach’s alpha after dropping a single item was 0.75 leaving 18 remaining items.
Face and content validity
A high level of staff involvement throughout the process of measure development ensured good face and content validity. This was achieved because staff participants provided feedback on the content of the themes arising from the qualitative data and on the language used in the item generation phase. Staff agreed that the results did capture what they had reported. The use of a flexible topic guide/interview schedule maximised exploration of the construct under study and minimised omissions in the data set.
Criterion validity
Those with negative perceptions of the barriers to change also had poor job satisfaction after controlling for age and occupational status (coefficient 10.43, SE 1.97; 95% CI 6.58 to 14.30, n = 101, eight wards; p = 0.001). The predicted mean VOCALISE score in the high job satisfaction group was 56.76 and the predicted mean score in low job satisfaction group was 67.20 ().
Do demographic characteristics affect perceptions of barriers to change?
Two demographic factors, age (coefficient –5.67, SE 2.57, 95% CI –10.70 to –0.64; p = 0.03) and occupational seniority, significantly affected staff perceptions of barriers to change when assessed using the total score.
Discussion
Using a participatory methodology, we have developed a staff generated, self-report measure of perceptions of barriers to change in acute care – VOCALISE. It demonstrates promising psychometric properties. The internal consistency and criterion validity are high and test–retest data show stability over time. The full involvement of nursing staff throughout the development of the measure has ensured that VOCALISE has good face and content validity and is accessible to the intended recipients.
In this study, staff had slightly more positive perceptions of job satisfaction than staff in other studies.101–103,200 Staff with negative perceptions of barriers to change also had poor job satisfaction. In previous studies, work stressors have differed between groups including occupational status and gender. Staff with higher organisational status52,54 view changes more favourably than those in more junior positions. Furthermore, job satisfaction has been linked to staff who express more open views about new changes.199 These findings were replicated with VOCALISE.
VOCALISE scores were most negative overall in the direct care staff groups (staff nurses/nursing assistants) compared with more positive scores in those who occupied more managerial roles (clinical charge nurses and team leaders). Both nursing assistants and staff nurses spend the most time in direct client contact. Therefore, their more negative perceptions might be one of the inevitable aspects of delivering therapeutic acute care for patients with severe mental illness at their most distressed. Furthermore, staff in leadership roles are more likely to be involved in the planning stages of new changes and, therefore, have an increased sense of control and responsibility over them, while those in direct care roles are more likely to be involved in delivering changes. These important issues, revealed through stakeholder involvement, require further exploration to discover key drivers for these perceptions which might then be subjected to management and other interventions.
The benefits of the participatory model used to develop VOCALISE are visible in both the breadth of the construct investigated and the rich content of the items. VOCALISE captures the complexity of an abstract topic by addressing the interaction between perceptions of the self and the organisation as well as perceptions of the team dynamic in relation to changes. Barriers arising from ward/environmental factors as well as from social factors are addressed. The content of some items is of interest because they exemplify feelings of organisational unfairness, perhaps as a result of poor consultation around new changes which may have been missed using a non-participatory method.
