Abstract
Background
In England, out-of-hours GP services provide urgent medical care to patients when their GP surgeries are closed. National Quality Requirement 5 (NQR5) requires out-of-hours services to routinely audit patient experiences, but provides no guidance on methods. In the absence of comparable data from providers, the out-of-hours items from the national GP Patient Survey have been used to monitor patient experience.
Aims
The aims of this study were to (1) explore whether variation in service users’ experiences of care were driven by user or provider characteristics, (2) document the validity of out-of-hours GP Patient Survey items and (3) understand how providers collect/use patient feedback to drive service improvements.
Methods
This was a multimethod study, analysing out-of-hours items from the GP Patient Survey data set (2012/13; 971,232 service users) and a bespoke survey (six providers; 1396 services users) and including a qualitative interview study with staff (11 providers and 31 staff).
Findings
Service users provided less positive ratings of out-of-hours care provided by commercial organisations than for out-of-hours care provided by either NHS or not-for-profit providers. Service users whose ethnic origin was ‘non-white’ or those finding it difficult to take time off work to attend their general practice also reported poorer experiences. GP Patient Survey data, subject to minor modifications, appeared valid and thus suitable for benchmarking. However, the items need updating to reflect the changes made to accessing out-of-hours services by telephone. Patient feedback (including GP Patient Survey data) has a limited role in driving changes to out-of-hours service provision, in part because of the lack of clarity of NQR5.
Conclusions
Out-of-hours items on the GP Patient Survey require refinement but appear suitable for benchmarking purposes. NQR5 is ambiguous and requires revision to assist providers in collecting and acting on patient feedback.
Introduction and rationale
Defining out-of-hours GP care
In England, out-of-hours GP services provide urgent medical care to patients when their GP surgeries are closed, that is, between 18.30 and 08.00 on weekdays and at weekends and bank holidays. Although medical care is largely provided by GPs, nurses and emergency care practitioners may also provide clinical care. Out-of-hours services are provided to manage health problems that cannot wait until the next working day; these services are not intended as an alternative route to health care for non-urgent problems for those patients who cannot attend during practice opening hours. Recent English national audit data reported that out-of-hours GP services handled around 5.8 million contacts during 2013–14, of which 3.3 million were face-to-face patient consultations.269
The provision of out-of-hours GP care has changed significantly in England over the last decade. In 2004, responsibility for out-of-hours services transferred from local GPs to NHS primary care commissioners. Commissioners are now responsible for purchasing care from provider organisations and, in some regions, very different models of care have emerged. In England there are currently 211 clinical commissioning groups269 commissioning out-of-hours services, although the number of providers is smaller, as many providers contract with two or more neighbouring commissioners. Out-of-hours services are also provided by different types of organisations, including NHS trusts, not-for-profit providers (e.g. social enterprises) and commercial health-care providers.65,270 Such services continue to evolve; since the phased introduction of the NHS 111 service, completed in February 2014,269 different providers may provide different aspects of care, for example call handling and delivery of clinical care, in the same geographical area.
Ensuring quality and safety of out-of-hours care
Although the reorganisation of out-of-hours GP care has the potential to bring about new approaches and increased efficiency of service provision, such reconfiguration may also generate reduced service coverage or quality. To tackle concerns regarding the quality of care provided, national standards were published with which all out-of-hours GP care providers were expected to comply.66 Providers are required to report their performance to their commissioners across a range of NQR recommendations. Of particular relevance to the IMPROVE programme is recommendation 5 (NQR5), which mandates out-of-hours providers to regularly audit a random sample of patients’ experiences and to take appropriate action based on the results.
Despite the introduction of the NQRs, criticism of the quality and safety of out-of-hours care persists.65,271 Prompted by the death of a patient in 2008, the CQC investigated the case and produced additional recommendations for commissioners and providers of out-of-hours services regarding performance assessment.270 More widely, urgent care provision in England has been criticised regarding service accessibility, the lack of continuity of care and concerns about patient safety.272–275 Within this context, the CQC has recently assumed responsibility for regulating and inspecting the quality and safety of out-of-hours primary care services.276 With CQC inspections commencing in October 2014, the latest CQC overview reported that the majority of service provision was of high quality, but that there were some areas in which improvements could be made.277
Role of patient experience surveys in quality assessment
Since 2015, service commissioners are expected to publish annual data on provider performance against the NQRs.278 Such a requirement is problematic for NQR5, as there is no agreed methodology for conducting patient experience audits. Without reliable and valid methods of assessing patient experience, it is impossible for providers to accurately assess their own performance and to subsequently use this information to guide service improvement. Providers may also use different tools and survey methods and the resultant data cannot be used for the purposes of benchmarking to assess variations in service quality between providers. Although a number of standardised patient questionnaires are available to assess patient experiences of out-of-hours primary care services,279 these tools have not been widely adopted in routine practice.
Although it is not possible to benchmark out-of-hours providers using the patient experience data collected for NQR5, the 2014 national audit of GP out-of-hours care269 and the CQC both analysed patient experience data from the English GP Patient Survey. The GP Patient Survey includes six items relating to out-of-hours care (two ‘access’ and four ‘evaluative’ items). As the only large-scale population survey of patients’ understanding, use and experiences of out-of-hours care, benchmarking of GP Patient Survey data is potentially possible. Establishing the validity of the GP Patient Survey out-of-hours items is, however, an important prerequisite to using this data to document variation in scores between out-of-hours services and for benchmarking. We have previously published evidence to support the reliability of the GP Patient Survey (including out-of-hours items).141 Using a range of different methods and analytical approaches, we have also demonstrated the validity of GP Patient Survey items evaluating in-hours primary care services,131,280 but this has yet to be established for out-of-hours care items.
Once the causes of poor patient experience of out-of-hours care have been understood, interventions to improve care can then be designed. However, the current literature on the effects of feedback of patient assessments is insufficient in scope, quality and consistency to design effective interventions targeting service delivery and organisation or the performance of clinicians.21,83,281
Rationale for the out-of-hours research
This research was designed to address these gaps in our knowledge to enable managers, patients and professionals to have confidence in the meaning of patient assessments of out-of-hours primary care services recorded in the national GP Patient Survey. The work package addressed three important areas.
The first workstream built on earlier analysis of the GP Patient Survey, which reported that important sociodemographic variations exist in patient experiences of in-hours primary care services,131 but did not examine if such variations existed for out-of-hours items. Given that the CQC and National Audit Office have both used the GP Patient Survey to monitor service users’ experiences of out-of-hours care, it is important to understand whether or not variation in service users’ experiences of care is driven by user characteristics, as opposed to differences in the care provided by different types of providers.
The second workstream sought to explore the validity of the out-of-hours items from the GP Patient Survey. The Out-of-hours Patient Questionnaire (OPQ) is a complementary tool to the GP Patient Survey, which collects more detailed information on patient experience of out-of-hours care and has undergone more extensive testing and validation.8,90,282 The second workstream tested the performance of GP Patient Survey out-of-hours questions against data derived from the OPQ to examine the validity of GP Patient Survey items.
The third workstream examined how out-of-hours GP services make sense of the information provided by patient questionnaires and, when possible, use this information to design interventions to improve patient experience through service reconfiguration and development.
Structure of the out-of-hours work package
The out-of-hours work package consisted of three workstreams, each of which used different data sets and methods. The remainder of this chapter describes the study aims and objectives, methods, results and discussion arising from each of the three workstreams in turn, before summarising the key conclusions that arose from the work programme.
Stakeholder advisory group
A stakeholder advisory group composed of three representatives from out-of-hours service providers, two primary care academics and a service user was convened to support workstreams 2 and 3. The group met to review study methods and procedures in light of the findings of preliminary piloting and testing of the methods (see Workstream 2) and to comment on topic guides supporting interviewing in workstream 3. Because of the logistical challenges of organising face-to-face meetings around staff availability, after an initial face-to-face meeting most advisory group input was secured by e-mail communication and telephone.
The original aim was to recruit two service users through our links with local service providers and using methods recommended by our Exeter University-supported PPI groups [see http://clahrc-peninsula.nihr.ac.uk/patient-and-public-involvement-in-research and www.folkus.org.uk (accessed 13 December 2016)]. Potential service user participants were provided with a brief information sheet regarding what would be involved in advisory group membership and were informed that any costs incurred in preparing for or attending advisory group meetings would be reimbursed. Despite significant efforts to secure lay stakeholder participation, it proved difficult to recruit service users with relevant, lived experience to the advisory board. Although this was problematic to the research, provider staff members indicated that their services experienced similar problems, probably because of the nature by which patients consulted (i.e. relatively infrequent consulters seeking care for an urgent problem) and the lack of continuity between provider and service user.
Changes to study methods from the original protocol
The overall aim of this strand of work, as stated in the original protocol, was to investigate how the results of the GP Patient Survey can be used to improve patients’ experience of out-of-hours care (aim 7).
In our original application we specified four objectives within this work package, three of which were successfully addressed within this programme (objective 1: cognitive testing of GP Patient Survey out-of-hours items; objective 2: establishing GP Patient Survey item validity and reliability; and objective 3: identifying how data from the GP Patient Survey can be effectively used to inform out-of-hours service reconfiguration). Objective 4, undertaking preliminary piloting of an intervention to improve patient experiences of out-of-hours care, was not achieved. The qualitative research undertaken to address aim 3 identified significant heterogeneity in terms of how providers collected and acted on patient feedback and in terms of the perceived utility of the GP Patient Survey as a platform on which to mount quality improvement. It was clear on completion of the qualitative work with service providers that more research was needed to design and then test the feasibility and acceptability of an intervention to embed patient feedback within quality improvement cycles.
For the three objectives that were achieved, some minor modifications to the study methods were implemented as the full protocols were developed. For example, it was initially proposed to interview up to 45 patients to test user responses to out-of-hours GP Patient Survey items. In reality, only 20 service users underwent cognitive interviewing, as this proved sufficient for testing the validity of the items. Similarly, to address objective 3, a more ambitious, qualitative interview study was undertaken with staff members from out-of-hours services. Here, 11 English providers (rather than six) were sampled and interviewed to ensure greater diversity in the types of provider organisation and the populations served.
Workstream 1: exploring variations in national GP Patient Survey out-of-hours items
Study aims and objectives
This workstream investigated:
potential associations between service users’ evaluations of out-of-hours GP care and individual-level sociodemographic factors
whether or not variations in evaluations were related to ‘clustering’ of service users reporting poorer experience within providers reporting poorer performance overall
whether or not there was an association between service users’ evaluations and type of provider organisation (NHS, commercial or not-for-profit organisations).
To address these aims, an analysis of service users’ ratings of out-of-hours GP care from GP Patient Survey data was undertaken.
Methods
Patient questionnaires
GP Patient Survey data (July–September 2012 and January–March 2013) were analysed [overall response rate of 35% (971,232/2,750,000)].283 The GP Patient Survey included four evaluative questions on out-of-hours provision, three of which were analysed: ‘timeliness’ of receiving care (‘about right’, ‘took too long’ or ‘don’t know/doesn’t apply’), ‘confidence and trust’ in the out-of-hours clinician (‘yes, definitely’, ‘yes, to some extent’, ‘no, not at all’ or ‘don’t know/can’t say’) and ‘overall experience’ of the out-of-hours GP service (five-point Likert scale from ‘very good’ to ‘very poor’). These questions were completed only by service users who had attempted to contact an out-of-hours GP service within the preceding 6 months.
Service user characteristics
Five sociodemographic variables derived from GP Patient Survey responses were analysed: gender (male as reference), ethnicity (white as reference vs. five categories derived from ONS data284), age in eight categories (18–24 years as reference), parent status (non-parent as reference) and whether the service user was able to take time away from work to attend his or her practice during working hours (individuals ‘not in paid work’ as reference vs. ‘paid work, can take time away’ or ‘paid work, could not take time away’). A sixth sociodemographic variable, deprivation (national IMD fifths; ‘least deprived’ as reference), was determined based on the respondents’ residential postcode.285
Practice and out-of-hours general practitioner service providers
Each service user was mapped to the out-of-hours GP provider responsible for providing clinical care for the service user’s practice during the 6-month period prior to sending the questionnaire. Mapping was achieved for 96% (934,931/971,232) of service users in the data set; 7886 practices were mapped to 91 out-of-hours GP providers, of which 86 had an identifiable provider organisation type (not-for-profit as reference vs. NHS or commercial).
Statistical methods
Analyses were performed using Stata 12. Sociodemographic data are described for all service users contacting an out-of-hours GP provider in the previous 6 months (for themselves or on behalf of another person). To facilitate comparison between measures on different scales, outcomes were linearly rescaled from 0 to 100,131 with a difference of < 3 points considered ‘small’ in respect of practical significance.102 Missing data at the level of service users or providers (including ‘don’t know’/’does not apply’) were excluded from the analysis. It was assumed that service user responses would be ‘clustered’ by out-of-hours provider (not practice), with clustering adjusted for as a random effect.
Three statistical models were employed. Model A was a fixed-effect multivariable linear regression model including individual sociodemographic factors as covariates and generated mean differences in outcome scores for comparator sociodemographic groups compared with reference categories, without accounting for differences in outcome across providers. Model B was a mixed-effects model that extended model A by incorporating a random intercept for provider. Model B therefore adjusted for differences in outcome between providers and estimated the mean difference between the comparator group and the reference group in outcome scores within providers. Comparing models A and B identified the extent to which any overall difference between service users of specific sociodemographic groups was due to clustering of service users within providers achieving a low outcome score.131
Model C extended model B by adding ‘provider type’ as a covariate. This model estimated the effect of provider type, with adjustment for service user characteristics, for each outcome. Comparing the between-provider variance from models B and C quantified the degree of between-provider variation attributable to provider type. The effect of provider type, analogous to an effect size such as Cohen’s d, was expressed as the standardised mean difference (mean difference between comparator provider type and not-for-profit providers divided by the between-provider SD derived from model C).
Results
The sociodemographic characteristics of 106,513 service users (from 7492 practices) who had contacted an out-of-hours provider and were mapped to a provider of a known organisation type are shown in . Service users’ overall evaluations of out-of-hours GP services were generally positive (): 71% (73,983/103,523) of participants reported a ‘very good’ or ‘fairly good’ overall experience, although 31% (31,966/104,145) felt that it took too long to receive care.
Sociodemographic characteristics of service users contacting an out-of-hours GP provider (on their own behalf or for someone else) (N = 106,513)
Timeliness of care, confidence and trust in out-of-hours clinician and overall experience of care: raw scores
Data were included for 86 providers: 44 not-for-profit, 21 NHS and 21 commercial providers. Provider type was associated with all three outcomes (global p-value < 0.001 for confidence and trust and overall experience, p-value = 0.013 for timeliness). No statistically significant differences were observed between NHS and not-for-profit organisations with regard to any of the outcomes, whereas commercial providers scored lower than not-for-profit organisations for all three outcomes (). The magnitude of these differences was approximately 3 points (model C) for all outcomes.
Associations of out-of-hours GP provider type with timeliness, confidence and trust and overall experience of care
A comparison of the between-provider variance (model B vs. model C) for overall experience of care observed that 18.6% of the between-provider variability was the result of provider type ( and see ); the equivalent values for timeliness and confidence and trust were 11.3% ( and see ) and 20.9% ( and see ). The standardised mean difference for commercial providers compared with not-for profit providers was –0.68 SDs for timeliness, –1.04 SDs for confidence and trust and –0.94 SDs for overall experience (see ). This equates to a moderate (timeliness) or large (confidence and trust and overall experience) effect size attributable to commercial provider type.
Overall experience of out-of-hours GP services: linear regression modelling
Timeliness of care from out-of-hours GP services: linear regression modelling
Confidence and trust in out-of-hours clinician: linear regression modelling
Service users of mixed ethnicity and Asian ethnicity reported poorer care for all three outcomes than white respondents; a more variable pattern of care was evident for service users of black ethnicity and other ethnicity (see –). In general, the mean differences in scores between white service users and service users from the mixed, black and other ethnic groups tended to be of lower magnitude that those between Asian and white service users.
A comparison of models A and B indicated that, with regard to timeliness, only 17% of the mean difference in scores between Asian and white service users derived from model A (–13.27, 95% CI –14.51 to –12.03; see ) was due to clustering of Asian service users within providers that scored lower overall (vs. 28%, 26% and 22% for mixed, black and other ethnicity service users, respectively). For overall experience of care, 35% of the mean difference between Asian and white service users derived from model A (–5.61, 95% CI –6.32 to –4.90; see ) was attributable to clustering of Asian service users within a lower-scoring provider.
Service users who could not take time away from work to attend their practice reported lower mean scores across all three outcomes than those for whom this was not applicable, whereas service users who could take time away from work reported higher mean scores (see –).
Other individual-level sociodemographic characteristics (gender, age, deprivation and parent status) were also associated with the three outcomes measures (deprivation was associated only with trust and confidence and overall experience) but the effects were not explored further because of the small magnitude of the mean differences when compared with the relevant reference category or because of more positive scores in the comparator category (i.e. potentially more disadvantaged) than in the reference group.
Discussion
Analysis of GP Patient Survey data identified that commercial provider organisations were associated with poorer reports of care across all three outcome measures when compared with not-for-profit organisations after controlling for patient-level sociodemographic characteristics. The lower scores associated with commercial providers is consistent with observations from US data showing that for-profit hospitals were associated with worse patient experiences than non-profit hospitals.286,287 However, the reasons underlying the lower scores for commercial organisations, even after controlling for individual sociodemographic variables, are unclear. This may reflect a genuinely poorer experience of care provided by commercial providers or the willingness of commercial providers to operate in areas deemed less attractive to NHS or not-for-profit organisations. It may also be that service users’ perceptions of provider type influenced their ratings, although it is questionable whether or not service users are aware whether their provider was a commercial organisation as opposed to a NHS or not-for-profit organisation, except perhaps in areas where media attention has focused on their local service.
Service users from minority ethnic groups tended to report less favourable care than white service users, with some variation observed across out-of-hours providers. This finding was in part attributable to clustering of minority ethnic service users in out-of-hours GP services with lower overall scores. Previous analysis of GP Patient Survey data regarding ‘in-hours’ care has indicated that minority ethnic patients reported generally lower experience scores131 and that patients of different ethnic backgrounds may differ with regard to drivers of satisfaction.102 In our analyses, although Asian service users reported lower mean scores than white service users for all three experience outcomes, the greatest difference was in the timeliness of care. Similar differences were seen for other ethnic groups, but of a lesser magnitude, suggesting that service users from minority ethnic groups, and Asian service users in particular, place substantial value on the timeliness of out-of-hours care. The ability of an out-of-hours GP service to meet service users’ expectations has previously been argued to be a strong driver of satisfaction with care,288 although this cross-sectional analysis cannot definitively answer this question.
Those who were unable to attend their practice because of work commitments were significantly associated with lower scores across all three outcomes than those not in paid work, whereas individuals who reported being able to take time off work reported somewhat better experiences. One explanation is that out-of-hours providers, who do not provide routine ‘non-urgent’ care, may not meet the expectations of service users who find it difficult to attend their practice during regular hours. However, as no information on the nature or the urgency of the service users’ health conditions was available this question cannot be addressed definitively.
Strengths and limitations
Unlike CQC and national audit data, this analysis of GP Patient Survey data was the first to map the majority of practices (and hence service users) to a specified out-of-hours GP provider and to determine the organisational provider type. The large sample available enabled sophisticated modelling to test the associations between provider and service user sociodemographic characteristics and service user evaluations of care.
Several limitations were evident regarding the data available from the GP Patient Survey. Service users were invited to provide feedback on their experiences of out-of-hours care in the preceding 6 months. Recall bias cannot be discounted, as previous research has found that older patients may not accurately report health service resource use over the short time frame of 3 months.289 No data were collected regarding the nature/urgency of the service users’ complaints, the time/date of the contacts or how the contacts were managed. Although data on ethnicity were collected, the GP Patient Survey did not ask about service users’ English language ability, nor about educational attainment, both of which may be related to experience of care.80 The lack of detailed response options regarding whether or not the service user was able to take time away from work and the timeliness of care also restricted our ability to interpret these data.
The GP Patient Survey response rate of 35% is also problematic. However, no evidence of an adverse association between response rate and non-response bias has been found for the GP Patient Survey and previous research using rigorous probability sampling methods (as used in the GP Patient Survey) has observed only a weak association between non-response rates and non-response bias.133,187,290 An analysis of data on out-of-hours care in the Netherlands suggested that non-response bias was small in respect of overall satisfaction with out-of-hours care.291
Workstream 2: establishing the validity of GP Patient Survey out-of-hours items
Study aims and objectives
The overarching aim of this workstream was to establish the validity of the GP out-of-hours care items within the GP Patient Survey to inform its suitability for benchmarking providers. This was achieved through a multimethod project composed of two stages. In the first stage, preliminary psychometric testing of the out-of-hours items was undertaken through cognitive interviews, combined with a pilot survey of out-of-hours users to test survey methods. The second stage tested the hypothesis that the GP Patient Survey items (modified after piloting) would demonstrate construct validity if together the GP Patient Survey items were correlated with the two known subscales of the OPQ (an established, valid and reliable measure of patient experience8,282). Concurrent validity would be established if the thematically relevant OPQ items were found to be associated with each of the GP Patient Survey items in linear regression modelling.
Methods
Settings
Six out-of-hours providers across England were recruited for a cross-sectional survey of service users. Data from year 5, quarter 2 (July to September 2010) of the GP Patient Survey11 were used to sample providers to ensure that there was variation in respect of performance (high/medium/low scoring) on respondents’ overall ratings of care received by GP out-of-hours services, as well as the type of provider (NHS, commercial, social enterprise) and the geographical area covered by the service (inner city/suburban, rural). Two participating service providers were operated by NHS trusts, three were operated by commercial companies and one was a not-for-profit social enterprise.
Survey piloting and cognitive interviews
A pilot study was conducted with two providers, with study questionnaires distributed to 500 service users (n = 250 per provider). Cognitive interviews with out-of-hours service users were conducted to explore the cognitive challenges faced by service users when completing the GP Patient Survey out-of-hours items and establish the validity of the item set. Twenty service users (predominantly female and aged ≥ 65 years) from two out-of-hours providers were interviewed using a think-aloud and four-stage verbal probing approach.292 Interviews were audio recorded, transcribed verbatim and analysed using protocol analysis.292
This preliminary work highlighted issues with the GP Patient Survey questions and with sampling of service users. The GP Patient Survey filters respondents to the out-of-hours items if they report having tried to make contact with a GP out-of-hours service in the previous 6 months, either for themselves or for someone else. As the respondents in this study were sampled from known users of out-of-hours providers, respondents were requested to evaluate their experience of the last time they made contact with a GP out-of-hours service. Minor modifications to the wording of the GP Patient Survey out-of-hours items (one item) and/or response options () and sampling exclusion criteria were suggested by the study team. These changes were reviewed and approved by the study advisory group prior to commencing data collection.
Changes made to GP Patient Survey items evaluating out-of-hours care following cognitive interviews with service users
Description of the questionnaire
The questionnaire had two sections. Section 1 contained the four modified GP Patient Survey evaluative stem items (applicable to all participants). These four items assessed service users’ ratings of the ‘entry access’ to the service, the ‘timeliness of care’ received, their ‘confidence and trust’ in the health professional who they consulted with and their ‘overall experience’ of the out-of-hours service. Section 2 consisted of the OPQ, which is composed of seven sections designed to capture information on the entirety of service users’ experience of out-of-hours care. The composition of the OPQ has been detailed elsewhere8 and was found to be both valid and reliable. Participants’ ratings on 14 evaluative items were analysed (); these were not management specific and assessed users’ experience of entry to the service, the outcome of their call and the consultation with a health professional.
The OPQ: 14 items used in analyses
Sampling
Sampling took place within 2 weeks of the service user contacting the out-of-hours service. The contact and demographic details for a random sample of 2000 service users were extracted from the electronic records at each site. Exclusion criteria were age 12–17 years, because of the risk of breaching patient confidentiality if a questionnaire was sent to a patient’s home address and because the GP Patient Survey targets those aged ≥ 18 years; admission to hospital as a result of the contact; palliative care needs; or a temporary/incomplete address. After all exclusions were applied, a questionnaire, accompanied by covering letters from the research team and service provider, an information sheet and a prepaid envelope, was sent to a consecutive sample of the first eligible 850 service users (or a parent or a guardian if the service user was a child) from the sampling frame at each site. In one area, only 818 service users were sampled because of logistical constraints in the screening process. The total sample approached therefore totalled 5068 service users. A reminder was sent 2 weeks after the initial mailing to non-respondents. Implicit consent was assumed if a completed questionnaire was received by the research team; no reminder was sent to service users who returned a blank questionnaire. Data collection took place between September 2013 and July 2014.
Data analysis
Respondents were compared with non-respondents with respect to their age, gender, deprivation quintile (using service users’ postcodes to derive their IMD285) and management option received as a result of the last recorded contact (from the service provider record: telephone advice, treatment centre attendance, home visit) using a multilevel logistic regression model, clustering respondents by the provider from which they were sampled.
Construct validity
Construct validity of the four modified GP Patient Survey items was assessed by ascertaining how well they summarised the OPQ. First, a confirmatory factor analysis was conducted to establish whether or not the OPQ possessed the same two-factor structure reported in the paper detailing its development.8 The standardised factor loadings with 95% CIs for this model are reported. As Hu and Bentler293 suggest, goodness of fit of the model was assessed through a two-index strategy using the standardised root-mean-squared residual supplemented with the comparative fit index (CFI),294 neither of which are adversely affected by large sample sizes.295
A principal component analysis (PCA) of the four modified GP Patient Survey items was then conducted to establish their latent structure, using the polychoric correlation matrix to account for the ordinal nature of these items.296 Inspection of eigenvalues and component loadings were used to explore the underlying structure of the responses. Based on this PCA, the construction of scales using the modified GP Patient Survey items and their internal consistency (Cronbach’s alpha) was explored. Finally, the correlations between the scales constructed above and the factor scores from the confirmatory factor analysis of the OPQ were investigated to assess the extent to which the modified GP Patient Survey item set summarised the OPQ.
Consultation satisfaction scale
The OPQ includes nine items rating service users’ satisfaction with their consultation with an out-of-hours clinician (see ). These items were combined into a ‘consultation satisfaction’ scale, as suggested by the paper validating the OPQ,8 to avoid issues of multicollinearity in the regression models. To achieve this, each item was linearised to a 0–100 scale and respondents’ mean scores from the nine items were derived as their consultation satisfaction scale score, provided that they had answered at least four of the items. Finally, the scale was standardised so that the regression modelling would produce standardised coefficients.
Concurrent validity
To investigate the concurrent validity of the modified GP Patient Survey items, four multilevel linear regression models were constructed, with a separate model for each evaluative outcome. The covariates were the management non-specific items from the OPQ (see ), including the consultation satisfaction scale. Concurrent validity was considered to be established if each modified GP Patient Survey outcome was found to be significantly associated with thematically related items from the OPQ. Univariate analyses were undertaken first, with covariates being excluded from the final models if they were not associated (p < 0.10) with any of the four outcomes. All models controlled for service users’ age, gender, deprivation quintile and management option, as well as the type of provider contacted (NHS, commercial, not-for-profit), and were clustered by provider. Missing data were accounted for using multiple imputations. To ensure that the regression coefficients of the covariates were comparable across models, the four modified GP Patient Survey outcomes, which originally had differing response scales (see ), were standardised. Sensitivity analyses were conducted to test for a linear trend over the covariate rating length of time taken for a health professional to call back, modelling the data while excluding those who answered ‘not applicable’ (n = 192). All analyses were performed using Stata 13.
Results
Response rate and sample
Completed questionnaires were received from 1396 out of 5068 (27.5%) sampled service users. The multilevel logistic regression assessing response indicated that responders were older and more affluent (lower IMD score), but did not differ with respect to gender. Differences in response rates were also evident across the management options (). The response distributions for all variables of interest are displayed in Appendix 6 (see Table 81).
Characteristics of responders and non-responders (N = 5067)
Construct validity
Confirmatory factor analysis of the Out-of-hours Patient Questionnaire
The confirmatory factor analysis revealed that the data fit the proposed entry access and consultation satisfaction two-factor structure reported by Campbell et al.8 moderately well (), with a standardised root-mean-squared residual of 0.06 (values of < 0.08 represent good fit) and a CFI of 0.89, which is just short of the suggested cut-off of 0.90 for good fit.293 In line with Campbell et al.,8 the two latent variables were moderately correlated (r = 0.54, p < 0.001).
Confirmatory factor analysis of the OPQ
Principal component analysis of the modified GP Patient Survey items
In the PCA of the four modified GP Patient Survey items there was a single component with an eigenvalue exceeding 1.0 (eigenvalue of 2.78), which accounted for 69.5% of the variance in the data. Observed component loadings were 0.44 for entry access, 0.47 for timeliness of care, 0.51 for confidence and trust and 0.57 for overall experience. This component can be interpreted as overall satisfaction with out-of-hours care. A rotation was unnecessary, as a simple structure was obtained.
Informed by the PCA, we investigated the construction of an overall satisfaction scale using all four items. This scale was derived by summing the standardised items (to account for differing response scales) if responses were given to all items. The scale had acceptable internal consistency (α = 0.772). Excluding the entry access item suggested a very minor improvement in alpha (α = 0.777; see Appendix 6, Table 82).
How well do the modified GP Patient Survey items summarise the Out-of-hours Patient Questionnaire?
The overall satisfaction scale was highly correlated with the factor scores of both OPQ domains for entry access (r = 0.63, p < 0.001, r2 = 0.397) and consultation satisfaction (r = 0.66, p < 0.001, r2 = 0.440). These correlations were both stronger than the correlation reported between the two OPQ domains. When combined into a scale, the four modified GP Patient Survey items explained 39.7% of the variation in entry access factor scores and 44.0% of the variation in consultation satisfaction factor scores, summarising both scales moderately well. reveals that the entry access domain of the OPQ was most related to service users’ experience of the call operator, for which there is no equivalent GP Patient Survey item, perhaps explaining the lower correlation between the overall satisfaction scale and the entry access factor scores.
Concurrent validity
Multiple imputation of missing data allowed for inclusion of all 1396 respondents in the four mixed-effects multilevel linear regressions. A divergent pattern of associations across the covariates was evident between the models for each of the four GP Patient Survey outcomes ().
Linear regression models showing the associations of OPQ items with the four modified GP Patient Survey outcomes
Discussion
This study sought to determine the construct and concurrent validity of four items from the GP Patient Survey evaluating service users’ experience of out-of-hours care through comparisons with an established, valid and reliable measure, the OPQ.8,282 Preliminary work highlighted the need to make minor modifications to three of the four GP Patient Survey items to improve comprehension by service users’ and response options. The modified GP Patient Survey item set (entry access, timeliness of care, confidence and trust and overall experience) formed a single scale, which summarised the two-domain structure of the OPQ moderately well. Therefore, given minor modifications, these findings indicate that the GP Patient Survey item set evaluating out-of-hours care has potentially acceptable construct validity as a scale of overall satisfaction.
Each of the four outcomes was strongly associated with a distinct set of thematically related items from the OPQ, demonstrating their concurrent validity. Evaluations of entry access were related to ratings of the length of time before service users’ calls were answered, the helpfulness of the call operator and the extent to which the operator listened, which is supported by these items loading onto the same construct in PCAs in the present study and elsewhere.8,130 Similarly, evaluations of timeliness of care were significantly associated with the time taken for the call to be answered, but were not related to ratings of the helpfulness of the call operator. Instead, timeliness was most strongly associated with the length of time taken for a call back from a health professional, an association also observed in a recent study of patient satisfaction with out-of-hours care from the Netherlands.291
Croker et al.148 found that patients’ confidence and trust in a health professional with whom they consulted in an in-hours primary care setting was highly influenced by interpersonal aspects of the care delivered as reported by patients. Important characteristics included having been given enough time, having felt listened to, having been given explanations about tests and treatments, having been treated with care and concern and having been taken seriously. In the present study, analogous items from the OPQ, combined into the consultation satisfaction scale, were strongly associated with service users’ ratings of confidence and trust in the out-of-hours health professional they consulted with. Confidence and trust were not related to items evaluating entry access.
Respondents’ ratings of their overall experience were strongly related to items from all three included sections of the OPQ: entry access, the result of the user’s call and the consultation with a health professional. The consultation satisfaction scale included an item rating the length of the consultation, which has also been shown to be a factor related to confidence and trust.297 Patients’ evaluations of their overall experience of in-hours primary care have been shown to be most associated with doctor communication and the helpfulness of receptionists.102 In the present study, service users’ ratings of their overall experience (the item unmodified from the GP Patient Survey) were strongly associated with their ratings of consultation satisfaction, which included elements of doctor communication as well as the helpfulness of the call operator.
Strengths and limitations
A strength of this study is the large number of service users included, which facilitated reliable statistical analyses using a large number of variables. When using factor analysis, best practice is to have five to 10 participants per measure,295 with a higher participant-to-measure ratio yielding more reliable results; upwards of 64 participants per measure were used in these analyses.
The overall response rate was low and responders tended to be older and living in less deprived areas; the final respondent sample also had a higher proportion of males than the non-respondent sample. This threat to the representativeness of the study sample is unlikely to have affected the analyses reported here. Specifically, this analysis aimed to determine the structure of users’ experience items and associations between them, rather than providing incidence/prevalence rates of conditions or similar outcomes that might be more affected by response bias issues. The methods employed controlled for these factors when possible and the findings are corroborated by the existing literature, as discussed above.
Minor modifications to either the word stems or response categories for three of the four GP Patient Survey items were made after careful piloting with service users that included the use of cognitive testing. Furthermore, the GP Patient Survey asks questions to respondents about making contact with a GP out-of-hours service in the past 6 months, whereas this study’s respondents were asked to answer questions relating to the last time that they made contact with a GP out-of-hours service, having been sampled from out-of-hours providers’ databases within 2 weeks of having made contact. Although this may limit the degree to which these findings apply to the existing GP Patient Survey items, this piloting was essential as early feedback from service users identified problems interpreting the items and changes to two items were designed to minimise missing data through blank responses (e.g. missing response categories). Implications for practice based on these findings are therefore contingent on the adjustment of current GP Patient Survey items.
Workstream 3: exploring how out-of-hours services use patient feedback
Study aims and objectives
This study aimed to identify how out-of-hours GP providers routinely collect patient experience feedback (including GP Patient Survey data) to inform their practice, with a particular focus on how it can be used to inform service reconfiguration and improve patient experiences of out-of-hours care. This was achieved by undertaking qualitative interviews with staff from out-of-hours service providers.
Methods
Sampling and data collection
The aim was to recruit an additional six out-of-hours providers as six (n = 12 total) were already recruited and had taken part in the survey study (see report on the conduct of workstream 2). Provider and staff recruitment ceased when data saturation was achieved. To achieve diversity of high-, medium- and low-scoring services, providers were first sampled on the basis of their scores on the GP Patient Survey item for care received from the service (question 40, April to September 2010 national GP Patient Survey data set). Once categorised into these groupings, information on organisation type and geographical location was considered. The final sample of providers ensured diversity across these three domains (GP Patient Survey score, organisation type and location), although no comparison of different subgroups of providers was planned. Up to three potential interviewees who had some involvement in conducting patient experience surveys were identified and approached to be interviewed at each provider. Participants were provided with an information pack consisting of a covering letter and participant information sheet. A mutually convenient time was organised to conduct the interview.
A week before the interview participants were sent a copy of a ‘feedback report’ containing patient ratings of their provider organisation based on the July 2012–March 2013 wave of the GP Patient Survey. Benchmarking data (generated by matching general practice postcodes to provider localities) were produced to allow providers to compare their performance with that of the 91 other English out-of-hours services for whom scores were able to be generated. Reports for the six services that had participated in the survey study (workstream 2) also included a summary of the their ratings derived from the research survey.
Face-to-face interviews, conducted at the participants’ workplace, took place between April and July 2014, each lasting between 39 and 88 minutes (mean 59 minutes). Topic guides were developed from a literature review, discussion between researchers and providers and previous findings with comments provided by the study advisory group. The topic guide included questions on how providers collected patient experience data and how this was used to make service changes; on awareness and views of the GP Patient Survey and out-of-hours items within it; and on the use of GP Patient Survey benchmarking provided in the feedback report.
Analysis
Interviews were digitally recorded and transcribed verbatim and transcripts were checked against the original recording for accuracy. Transcripts were coded in NVivo 10 software and analysis was independently coded using an iterative approach by one researcher (HB). A sample of five transcripts was independently analysed by a second coder (AA) to ensure that agreement was reached on the coding frame and codes. A deductive, framework approach with preliminary codes reflecting the content of the topic guide was used to construct the coding framework. However, a more inductive approach with additional thematic coding was undertaken using the ‘constant comparison’ method297 to capture new themes emerging from the data set. The initial coding frame was discussed within the team and when possible the codes were tested through seeking negative cases and/or divergent data. The data were then reorganised and collapsed into overarching themes. This process took place on two occasions until the main categories were agreed. All participants were sent a summary of the findings with a structured feedback form inviting comments on the veracity of the interpretation of the study findings. Final themes were reviewed and agreed between the research team to enhance reliability.
Results
Study participants
Five of the six providers approached took part (in addition to the six who participated in the survey study). A total of 31 staff from the 11 providers (NHS organisations, n = 2; social enterprises, n = 4; commercial organisations, n = 5) were interviewed, at which point data saturation was judged to have been achieved. Most participants were female (n = 23); 18 were service managers, seven were clinicians (GPs) and six were administrators. Participants who completed the feedback form (n = 2) on the findings were satisfied with the accuracy of the summary. Three main themes emerged: using surveys as a method of obtaining patient feedback; the utility of patient feedback; and the value of benchmarking.
Surveys as the most common method of obtaining patient feedback
Most participants focused on survey methods for collecting patient feedback, as 10 of the 11 providers undertook regular surveys to audit their patients’ experiences. Participants also discussed the ambiguities of operationalising NQR5, the desire for qualitative feedback to supplement survey data and the role of alternative methods in addition to surveys. It was evident from discussions that each provider interpreted the sampling for NQR5 differently, for example the range of patients being routinely audited varied from 1% to 20%:
We send out approximately 250 a week. Our National Quality Requirements require us to survey 1% – we actually do considerably more than that because we have taken our own interpretation on it.
11_4001, manager
Audits were undertaken on either a weekly or a monthly basis, using survey instruments developed by the organisation. Some participants reported that weekly audits were useful in terms of maximising patient response rates:
[T]hey’ve [out-of-hours service] worked out that the sooner the patient gets the questionnaire the more likely it is that they will complete it because it’s still fresh in their minds, so they try to do it as quickly as possible.
14_4003, GP
Most participants placed great importance on qualitative feedback from free-text comments provided by patients, which helped to interpret the quantitative findings, identify actions and provide a more personalised response from patients:
If they have got a real issue they can put it down, can’t they? Just doing the survey itself is just a way you test the water . . . The free text allows someone who has got a very bad experience the opportunity to write to us.
10_4001, manager
I’m dealing with people, I’m not dealing with robots. I mean, it’s their experiences, their feelings and they need to have a place to feed that back . . . they absolutely need to have a place to express their opinions – that’s giving people a voice.
14_4003, GP
Although it was agreed by all but one of the participating providers that patient surveys were a necessity, this was not a sufficient resource to drive change within services. A wide variety of alternative methods used by providers were reported, such as comment cards, ‘complaint and compliment systems’ and new technologies:
At the moment we’re thinking of going more electronically, so as soon as you have your consultation in the base, you come out and there’s a tablet so you can actually do your surveys straight after . . . that way you can get more accurate feedback of how people are feeling.
19_4002, administrator
Utility of patient feedback
Many participants cited examples of ways in which patients’ reported experiences had been used to make changes to service provision, although most changes tended to be ‘low level’. Because of the lack of observed trends within the data, most participants reported that patient survey data were insufficient to instigate service-wide changes:
In the main the results are stable and pretty good, but there’s not enough that’s consistent that I think we could use around wholesale service change.
12_4003, manager
Participants reported that patients’ expectations of the out-of-hours service were often unrealistic and difficult to manage and this made patient feedback difficult to deal with:
You often get patients who are very unhappy about the service they got and when you drill down into it it’s because they didn’t get antibiotics for their cold. Its expectations.
16_4003, GP
The changing landscape of the urgent care system was also confusing to patients. Some staff participants questioned the validity of patient experience data as the patients might be unaware of the different elements of the care pathway. Another barrier identified was the low-level engagement by commissioners. Despite the fact that patient experience audits are part of NQR5, many participants reported that commissioners treated them as a ‘tick-box’ exercise:
They [the commissioners] don’t come across to me as particularly engaged in this at all, and never really ask us too many questions around it.
18_4003, manager
Although acknowledging the identified barriers, some participants discussed how engaging with patient feedback had subtly changed the culture within their organisation and highlighted the importance of transparency and being responsive to change. In addition, participants reported the benefits of being able to compare patient feedback with other areas of reporting within the NQRs.
Value of benchmarking
Most participants acknowledged the benefits of having access to benchmarking data and felt that these data were a facilitator to enabling change. Notwithstanding this, many staff interviewees placed greater importance on their own surveys over the GP Patient Survey data, largely because their own surveys were more detailed.
Some staff expressed concerns about the reluctance of some providers to share with and learn from other providers, an issue mainly arising from commercialisation taking place within the NHS:
It’s terrible isn’t it, when everybody’s competing and not collaborating? That’s the system we’re living with, we’ve had to get used to it.
18_4001, GP
The benchmarking provided using the GP Patient Survey out-of-hours patient ratings was seen as useful, although many identified weaknesses with set items as they felt that the questions did not reflect the current urgent care system and lacked detail:
It is [GP Patient Survey out-of-hours evaluative items] just four questions, you get asked in McDonalds. It’s not detail is it?
10_4001, manager
Discussion
In the UK out-of-hours primary care providers are mandated to regularly audit patients’ experiences as part of the NQRs and services routinely meet this requirement by conducting patient surveys as well as by obtaining feedback using a variety of other methods. However, NQR5 is ambiguous and the resultant data cannot be used to compare services as providers are undertaking audits of varying scale, frequency and methodology. Staff reported a strong preference for qualitative patient feedback, which is echoed in other settings, as it yields richer, more detailed feedback than quantitative survey scores. For example, hospital staff have found that qualitative data from patients added a more patient-centred aspect to patient satisfaction measurements.210,298 Research has shown that health-care leaders place great importance on complaints, comments and compliments as sources of patient feedback,299 as do general practice staff (see Chapter 7).
Patient feedback appeared to have a limited role as a driver for service change and effective change was hindered by modifications taking place in the urgent care landscape, which confused patients with regard to how care was organised. Some staff also reported that commissioners appeared uninterested in patient experience audit findings. In some settings audit and feedback have been shown to have small to moderate effects on health-care professionals’ practice,204,300 although in other settings it can have a wider impact.301 For change to occur, the organisational culture must be supportive of change and be patient focused.206,209,302 Most of the changes reported by staff were ‘low level’ and unlikely to drive system-wide reconfiguration because of the lack of consistent patterns observed in the data. There was a preference for qualitative feedback as patient free-text comments could potentially identify specific areas of actionable change or contribute to wider data-gathering audits, for example critical incident techniques.303 However, to be useful, patients’ attention must be focused to provide qualitative feedback on the out-of-hours service.
Staff valued the GP Patient Survey patient experience benchmarking data and the GP Patient Survey presents an opportunity for benchmarking of all out-of-hours services. NHS England has recently recommended that NHS commissioners use the GP Patient Survey results to monitor patient experiences of out-of-hours providers278 and the CQC has published GP Patient Survey provider performance at commissioner level.277 Despite the strengths of the GP Patient Survey (regularly and independently collected data that is publicly available), participants were reluctant to use GP Patient Survey data in its present form because of concerns about the face validity of out-of-hours items and the absence of free-text comments, a limitation found in previous studies.119,199 In addition, the current out-of-hours items are not reflective of the recent changes that have taken place within the urgent care system (e.g. introduction of the NHS ‘111’ telephone portal). Most staff did not believe that the limited number of GP Patient Survey items would drive change by themselves.
Strengths and limitations
This is the first qualitative study to explore the views of out-of-hours staff who have an in-depth knowledge of patient feedback processes within their organisation. Sampling ensured that staff from a variety of different types of provider (e.g. not-for-profit or commercial enterprises), serving diverse populations across England, were included. Although sampling diversity was achieved, it is acknowledged that participating organisations may be more interested in the patient experience agenda than non-participants and thus findings may not reflect the views of the wider population. The views of commissioners were not sought in this study and thus the widespread perception that some commissioners were apathetic towards patient feedback data must be interpreted cautiously. Because of logistical constraints it was not possible to interview commissioners and obtain their perspective on the perceived role and value of patient feedback data.
Conclusions from the out-of-hours research
Implications for practice and future research
An analysis of national GP Patient Survey data (see Workstream 1) identified that commercial providers were associated with poorer patient experiences of out-of-hours GP care than NHS or not-for-profit providers. It is not possible to derive simple explanations regarding the drivers of these lower ratings in this observational data set and further research is required to understand what drives these differences. Although some insight might be gained from an understanding of patient differences (e.g. nature or urgency of requests for care) at the level of the provider, such data are not routinely collected in the GP Patient Survey for out-of-hours service evaluations. It is unknown whether or not factors such as user awareness of the provider type may also be of importance in interpreting service users’ ratings.
Further research, possibly involving qualitative approaches or a vignette study, is required to investigate the reasons for the generally lower scores from service users from minority ethnic backgrounds (see Chapter 6 for vignette work conducted as part of the wider IMPROVE programme). Similarly, research investigating the reasons why service users who were unable to take time off from work to attend their practice during regular hours reported poorer scores across all three evaluative questions is needed. Finally, as for in-hours GP care,131 investigation of the extent to which variations between sociodemographic groups in respect of care ratings might be attributable to the clustering of servicer users belonging to sociodemographic groups reporting relatively lower scores within providers with lower overall scores is required. This analysis would help inform the development and targeting of interventions aimed at improving service users’ experiences of out-of-hours GP care for specific population subgroups.
National standards (NQR5) require out-of-hours providers to routinely audit patient experiences, although no specific survey tools or methods are recommended to achieve compliance. In the absence of data collected directly by providers, both the National Audit Office and the CQC have recently used the GP Patient Survey as an alternative data source to monitor patient experiences of GP out-of-hours care. However, an important prerequisite to using GP Patient Survey data to benchmark services is that its psychometric properties are established. The reliability of GP Patient Survey out-of-hours items have been previously reported,141 but there was no evidence regarding their validity. The second workstream demonstrated that, although our survey was composed of only four of the GP Patient Survey evaluative items (after minor but essential modifications identified through cognitive testing and piloting), the GP Patient Survey out-of-hours items that we used had both construct and concurrent validity. These findings provide support for the use of the GP Patient Survey for national benchmarking purposes.
Whereas workstreams 1 and 2 examined the technical performance of the GP Patient Survey out-of-hours items, the third workstream examined how out-of-hours staff use patient feedback and their views on the utility of GP Patient Survey items. This qualitative study found that, although NQRs are intended to promote transparency and allow comparisons between out-of-hours providers, NQR5 was ambiguous and in its current form does not support benchmarking or service improvement. A critical review of the NQRs is required to help providers to engage with patient feedback and drive service improvement effectively.
In the absence of clear NQR guidance, providers were inventive in the ways in which they engage with patients. Qualitative feedback was highly valued as it provided detailed information that could lead to actionable changes. However, services struggled to find ways to use patient feedback to drive anything other than low-level service change. Future research should explore how out-of-hours services managing patients with urgent care needs, and particularly those delivering services to diverse populations, can be assisted in engaging more fully with patient feedback. Evidence is also needed on whether or not comprehensive guidance on how to collect, interpret and act on patient feedback has the potential to drive quality improvement initiatives.45,206,302
In the context of the rapidly changing landscape of UK urgent care services, although participating providers could see the potential of using the GP Patient Survey for benchmarking purposes, its out-of-hours items need urgent revision as they do not reflect current telephone access arrangements (NHS 111) for out-of-hours care. This qualitative finding supports our preliminary survey piloting work and cognitive interviews with service users (see Workstream 2). Minor but essential amendments to the GP Patient Survey out-of-hours items are required to improve the comprehension of items and improve data quality.
Patient feedback currently has a limited role in driving changes to out-of-hours service provision and the utility of feedback may be hindered, in part, by recent modifications to the urgent care system and the ambiguity of NQR5 in relation to gathering and acting on patient feedback. English GP Patient Survey data may be used to benchmark and compare service providers. However, the out-of-hours items need to be updated to reflect the changes made to accessing out-of-hours services by telephone, so that providers can be confident that ratings reflect their services’ performance. A greater understanding of how variations in patient and provider characteristics drive variations in patient experiences of out-of-hours care is needed to support the development and targeting of quality improvement initiatives.
