The project had two main strands – a survey of LAs and realist review – both supported by strong stakeholder engagement. This chapter begins with a section describing the role of our stakeholder groups throughout the project, before describing the methods used in our survey and realist review.
We conducted a survey of LAs with the aim of providing a comprehensive overview of how different localities across England implement the NHSHC programme. In addition, the survey aimed to gather additional material (including local knowledge, unpublished evaluations and examples of best practice and Covid-19-related innovation in delivery) for the review.
In addition, we conducted a realist review to enable us to better understand the important contexts that influence NHSHC delivery, and the mechanisms which produce both intended and unintended outcomes. Realist review is an interpretive, theory-driven approach to evidence synthesis that aims to develop an in-depth understanding of how, why, when and for whom complex interventions (such as the NHSHC) ‘work’. We chose this approach because existing research clearly demonstrates that the NHSHC programme is a complex intervention with context-sensitive outcomes, that is, the programme’s outcomes are highly dependent on the circumstances in which the NHSHC is delivered and received by attendees. There is significant heterogeneity in commissioning and delivery of the programme across England and wide variation in key outcomes, including rates of attendance, follow-up, and the provision of advice, onward referral and prescribing post-NHSHC.7,8 Our review sought to improve understanding of this variation, via developing a programme theory that identifies the important contexts and mechanisms that produce NHSHC outcomes, with a specific focus on what happens after the measurements and risk assessment are complete.
The full project plan is presented in . The project was registered with PROSPERO (CRD42020163822) in January 2020. The start date of the project was delayed to October 2020 due to Covid-19, and the project was conducted over 15 months, completed in December 2021.
Stakeholder groups
We recruited two stakeholder groups to provide us with content expertise and a range of perspectives throughout the project.
Throughout the project, our strategy for patient and public involvement (PPI) was informed by our PPI lead (VH). VH is an experienced PPI contributor (via the Public Involvement in Pharmacy Studies at Medway School of Pharmacy Group) and brought her valuable perspective as a member of the public and skills in group facilitation. In particular she supported us in developing lay summaries, advertising for further PPI contributors and supporting our PPI group throughout the project.
Our PPI group involved 10 members of the public from six different English regions, all of whom are eligible to receive the NHSHC.37 This group was recruited by advertising via the Oxford Biomedical Research Centre website and the University of Kent’s Centre for Health Services Studies’ Opening Doors to Research group. We offered PPI contributors a shopping voucher as a token of our appreciation for their involvement, and covered any expenses they incurred in taking part. From those members of the public who came forward, we purposely selected a group who were as diverse as possible in relation to gender, age, ethnicity and geographical location, with the aim of capturing a range of different perspectives from individual members of the public.
Our ‘professional’ stakeholder group were recruited via our project team’s existing networks and snowballing from these, and included LA commissioners, NHSHC providers, an NHSHC trainer and representatives from relevant health charities. In addition, we maintained close contact with PHE’s (latterly OHID’s) CVD Prevention Programme lead throughout the project.
Both groups were consulted via regular online meetings throughout the project, using Microsoft Teams (Version 1.0, Microsoft Corporation) and Zoom (Version 5, Zoom Video Communications, Inc.) video conferencing software. Our approach to facilitating our meetings was driven by the need to work to ensure that these online discussions – necessary due to the Covid-19 pandemic – were as open and inclusive as possible. Based on the recommendation of our PPI lead (VH) we met with our PPI and professional stakeholder groups separately, to help ensure that members felt able to speak openly about their experience of receiving and/or commissioning/providing the NHSHC and to reduce the number of people in each meeting, giving individuals more opportunity to contribute. However, CD attended all meetings of both groups and, where relevant, reported discussions from the PPI group to the professional group and vice versa. Our PPI stakeholder group meetings were chaired by VH and our professional stakeholder group meetings by GW. To facilitate more in-depth discussion and give each participant more time to contribute, our PPI group was split into two smaller groups for the last two meetings.
Our final professional stakeholder group meeting was augmented with additional contributions from 36 individuals. They came from a range of professional backgrounds (LAs, Public Health, OHID and providers) and from geographically diverse parts of England. Individuals were recruited via OHID’s existing networks. These additional contributions were sought to ensure that we would get broader feedback and advice on our findings and recommendations.
We consulted the stakeholder groups in relation to the focus of the project and asked them to provide feedback on our emerging findings as they developed. To help facilitate discussion on our findings, we circulated summary material before each meeting and provided an overview slide presentation at the beginning of each meeting. We initiated discussions with sets of simple questions to help guide the participants; for example, we asked stakeholders how our findings related to their own knowledge and/or experience of commissioning, providing or attending the NHSHC, and we asked them about important influences on NHSHC delivery or the response to the NHSHC that seemed to be missing from our findings. Our aim was both to identify those findings with particular resonance for stakeholders, but also to highlight any important aspects of NHSHC delivery that are not well represented in the literature. We consulted the professional stakeholder group on the development of our survey questions, to ensure their relevance and importance, and on our survey design, with the aim of maximising clarity, validity and the likelihood of achieving a good response with full completion of questions.
The discussions held during these meetings helped to shape our analysis as the project progressed. Input from both the professional and PPI groups influenced the analysis of our survey responses and our interpretation of the data included in the review. For example, input from the professional stakeholder group informed the development of a typology of NHSHC delivery models and the identification of relevant factors to consider in our analyses. In discussions of the review’s emerging findings, our PPI group members emphasised the need for NHSHC attendees to have more control over what happens during and after a check, especially in relation to the discussion of ‘lifestyle’ advice and opportunities for referral to other services. As a result, we reconsidered our data on attendee responses to what providers offer or discuss at the end of an NHSHC encounter, to consider whether it could tell us more about potential mechanisms producing these important outcomes for attendees. (see e.g. CMOCs A16–A17, A30–32 in Chapter 3). Similarly, PPI members’ repeated observations on the ‘mismatch’ of the focus of the NHSHC with their own health needs and priorities helped us to shape our findings (see e.g. CMOCs A9–A11) and recommendations in this area. Overall, our PPI group felt strongly that they needed more clarity on the purpose of the NHSHC programme and the group’s in-depth discussions on this point are represented strongly throughout our findings and in our subsequent recommendations for policy and practice (see Chapter 3 and Chapter 4).
In the last meetings with each group, we asked the participants for their input to help us to develop and refine practical recommendations for NHSHC delivery based on our findings and to inform our dissemination strategies, to help us to develop tailored outputs and identify the key ‘players’ for dissemination amongst different audiences. We refined our recommendations and developed plans for dissemination activities and project outputs on the basis of feedback from both groups.
A summary of the stakeholder group meetings and important input from these groups is provided in .
Summary of stakeholder meetings
Survey methods
Survey aims and objectives
The aim of the survey was to enable us to gather additional material (local knowledge, unpublished evaluations and examples of good practice and Covid-19-related innovation) for the project, and to provide a comprehensive overview of how different localities across England implement the NHSHC programme. The objectives were:
To describe how NHSHCs are delivered across England, particularly in relation to what happens after the measurements and risk assessment.
To determine how the Covid-19 pandemic has changed the way NHSHCs are delivered in some areas.
To categorise, as far as possible, different models of
NHSHC delivery employed across England.
To determine any associations between
NHSHC delivery models and other variables, based on relevant publicly available data.
Survey development and piloting
An online survey (available in full on our project website www.journalslibrary.nihr.ac.uk/programmes/hsdr/NIHR129209/#/) was developed using Jisc Online Surveys, chosen as it is designed to generate professional academic survey formats. The survey was to cover all LAs in England with responsibility for commissioning the NHSHC programme (i.e. upper-tier and unitary). Given the timing of the survey – during the second year of the Covid-19 pandemic, and less than a year since PHE’s own survey on NHSHC programme delivery – we were concerned that our survey would add an unwanted and unwarranted burden, unless it asked new questions and was able to provide new information. The survey was therefore designed to capture (1) changes in delivery in response to the pandemic and (2) detail about the delivery of the programme with a focus on what happens after the measurements and risk assessment are complete.
We also used the survey to find out whether LAs had commissioned, conducted or been part of any assessments of the NHSHC programme in the previous five years (including evaluations, collection of attendee feedback, health equity audits, or any other type of study), and to request copies of these to be included in our review.
The survey questions were designed in collaboration with our stakeholders, who helped us to ensure that the content of the questionnaire addressed the objectives of the research, the instructions and questions were clear and concise, and the survey was straightforward and quick to complete (i.e. in less than 15 minutes).
The survey included a mix of simple closed- and open-ended questions. The latter were included to enable respondents to both explain their responses to closed questions and add any further information about commissioning and delivery. Four-point rating scales were used to obtain level of confidence in capacity, accessibility and usage of support services (where 1 means ‘not confident at all’, and 4 means ‘very confident’).
The resulting online survey was sent to seven LAs (NHSHC/Cardiovascular Health leads) on 30 April 2021 to ‘test’ the system in advance of full roll-out. The survey was completed in full by three respondents, and feedback on the survey itself was positive. One subsequent change was made to the survey prior to the main launch. This was to include a single additional question at the start (Q3: Has the delivery of NHSHCs in your area resumed since the Covid-19 pause? Yes/No).
Administering the survey
To ensure a maximal response, and to ensure the survey was correctly targeted to those best placed to answer it, we worked closely with PHE to distribute the survey and make use of their tried-and-tested processes for dissemination. PHE sent the survey on our behalf to two of their governance groups: the PHE regional NHSHC Leads, and their Local Implementor National Forum, which includes LA
NHSHC implementers representing all of the PHE regions. These groups were then asked to disseminate the email to their networks (e.g. the regional leads were asked to cascade the message to contacts for the LAs in their area). PHE also publicised the survey through their established NHSHC programme webinar series. Following the launch on 17 May 2021, a first reminder was sent on 4 June, and another on 18 June (announcing an extension to the deadline). On 2 July, targeted reminders were sent to leads in regions with a response rate of less than 35% (West Midlands, North East, East of England and London), for further cascading to all LAs in their regions. The survey closed on 18 July 2021.
Data handling
Survey responses were recorded online then downloaded into Microsoft Excel and a statistical software package (IMB SPSS Statistics 27) to aid analysis. Respondents were allocated unique identifying codes (R01–R68). Respondents were asked to name the LA(s) on behalf of which they were responding. Where one respondent was responding on behalf of multiple LAs, responses to all remaining questions were copied for the relevant LAs. Each LA was then allocated a unique identifying code (LA01–LA74). Qualitative responses given in free-text boxes were used to clarify or amend responses where relevant. For a few responses that appeared confusing or limited, we conducted a search of that council’s website in order to double-check information given (e.g. with regard to where NHSHCs are made available).
Data analysis
Simple descriptive statistics were used to analyse quantitative responses to the questionnaire. Where relevant, qualitative responses were categorised to enable descriptive analysis; for example, the venues where the NHSHC was offered. Qualitative responses were also used to illustrate the quantitative data throughout, for example by contributing to case examples of delivery types.
In order to develop a typology of LAs based on delivery of the NHSHC, data from responses to several questions were combined. This enabled categories to be devised, based on how LAs were delivering NHSHCs before the Covid-19 pause and their use of remote methods after the pause. The following responses were used to categorise delivery models:
number and type of venues pre-Covid-19 pause
number and type of providers delivering face-to-face and/or remote health checks
number of remote methods used post-Covid-19 pause.
Based on their responses to these questions, LAs were grouped into three categories as outlined in .
Typology of NHSHC programme delivery models
Associations between category of delivery and other factors
Associations with the following other survey responses were assessed:
total number of commissioned services reported
total number of referral processes reported
total number of methods used to prioritise invitations pre- and post-COVID pause
average reporter confidence in capacity and accessibility of support services, confidence in their usage (rating scale)
total number of monitoring aspects plus evaluation reported (categorised as none, minimal, above average).
In addition, key statistics related to the public health function, populations, and performance measures for NHSHC programme delivery at LA level were obtained from publicly available data. For each responding LA we recorded:
geographic region:
PHE centres
38size: estimated population, 2019
39budget: Public Health budget per head, 2019/20
39deprivation: Indices of Multiple Deprivation (IMD) 2019
40NHSHC programme delivery performance: proportion of eligible people receiving an
NHSHC between 2015/16 and 2019/20.
41
On the request of our stakeholder group, we also explored the possibility of including the rurality index in our analysis, as an external factor that is likely to influence NHSHC delivery. However, this was found to be problematic since rurality is classified at district level, meaning no classifications are available for the large county (upper tier) councils (of which there are 19 in our dataset). In addition, due to the number of classifications (six), and the exclusion of the county councils, there were very few responding LAs within any of the more rural classifications.
One-way analysis of variance was used to compare service provision, referral processes and delivery performance across delivery categories. Chi-squared tests were used to assess associations between delivery category and other variables. Spearman’s correlation was used to assess relationship between variables.
Review methods
Review questions
What are the mechanisms by which the current
NHSHC programme produces its intended outcomes after the measurements and risk assessment?
What are the important contexts which determine whether the different mechanisms produce intended outcomes?
In what circumstances are such interventions likely to be effective?
Our realist review followed Pawson’s five iterative stages42 as outlined in our protocol.1 The steps we followed are summarised below.
Step 1: Locate existing theories
The goal of this step was to identify existing theories that explain when, how and for whom the NHSHC programme is supposed to ‘work’, to achieve its desired outcome of reducing CVD risk and mortality. The rationale for this step is that interventions and programmes like the NHSHC are underpinned by implicit and explicit assumptions and theories about how they should work in practice.43 To locate existing theories that might offer explanations for NHSHC processes and outcomes, we undertook two iterative processes:
We drew on the knowledge and experience of our own project team (including in general practice, pharmacy, public health and lived experience), and consulted with our stakeholders.
We informally searched the literature to identify:
existing theories, consulting both grey literature in the form of
NHSHC programme documentation, and published research studies that employed formal or substantive theories to understand the
NHSHCexisting reviews and evidence syntheses focused on the
NHSHC programme to develop our understanding of the existing research landscape and identify gaps in knowledge.
For step (b), we identified NHSHC programme documentation via searching and browsing the NHSHC website (www.healthcheck.nhs.uk) and the UK DHSC website (http://dh.gov.uk), and archived versions of these websites (via the UK Government Web Archive (http://nationalarchives.gov.uk) and the Internet Archive WayBack Machine (http://archive.org/web/)). We identified existing theoretical literature by running searches in PubMed and Web of Science (Core Indexes – SCI-EXPANDED, SSCI, ESCI, AHCI), using a slightly modified version of Booth and Carroll’s BeHEMoTh approach for searching for theory.44 We searched for the health context (the NHSHC) and terms relating to ‘theory’ and used citation tracking to identify additional studies. To identify published reviews, we consulted bibliographies produced by PHE, who have undertaken regular literature searches for evidence relevant to the NHSHC programme since 2015 (www.healthcheck.nhs.uk/commissioners-and-providers/evidence/literature-review/). The full details of the searches employed are available in Appendix 2.
Overall, we consulted 60 programme documents (policy papers and guidance) and identified 19 existing studies of the NHSHC that employed six formal theories or theoretical frameworks, and 10 existing reviews and evidence syntheses. Details of these documents are provided in Appendix 1.
In reviewing the existing research evidence relating to the NHSHC programme by consulting existing reviews and evidence syntheses, we identified an existing focus on the early steps of the initial programme theory (IPT) as outlined above, and especially on the processes involved in invitation and uptake of the NHSHC. Conversely, less research attention has been paid to later stages, and especially to what happens after measurements and risk assessment at the end of a check, in relation to advice, onward signposting and/or referral and ongoing support for lifestyle and behaviour change. As described above in Chapter 1 (see Project focus: what happens after the risk assessment in a Health Check?) we determined (in consultation with our stakeholder groups) to focus our own review on these later steps.
We combined our understanding of the programme from the literature reviewed with our own knowledge to develop and refine a coherent IPT for our realist review and to inform the subsequent stages of searching, data extraction and analysis. The IPT is presented in ; it maps out the steps involved in the delivery of an NHSHC and highlights our particular areas of focus and questions we considered in the course of the project.
Initial programme theory for the realist review with project focus highlighted.
In recognition of the complex nature of the NHSHC programme, and the possibility (captured in our IPT) that important interactions and feedback loops may exist with earlier steps in the delivery of the check, we determined that we should not exclude evidence relating to other programme steps. Instead, we decided to focus initially on identifying those documents that could provide data relating to the later steps, and subsequently draw on data that may shed important light on relationships with other steps in the programme as necessary.
Step 2: Search for evidence
The aim of this step was to identify a relevant body of literature containing data that could be used to develop and refine the IPT developed in Step 1.
At the outset of this project, we were aware that PHE regularly undertakes literature searching for new evidence relating to the NHSHC. These regular searches employ a comprehensive search strategy across 13 relevant sources (PubMed, MEDLINE, Embase, Health Management Information Consortium (HMIC), Cumulative Index to Nursing and Allied Health Literature (CINAHL), Global Health, PsycINFO, Cochrane Library, NHS Evidence, Google Scholar, Google, Clinicaltrials.gov and the ISRCTN registry). These searches have been used in previous review projects commissioned by PHE to identify and synthesise evidence relating to the NHSHC.7,8 These published reviews have included additional searches in OpenGrey and/or Web of Science (Science Citation Index, SCI-EXPANDED); the most recent of these PHE-commissioned reviews captured studies published until the end of December 2019.
As described in our protocol, we did not duplicate this existing work, but aimed to re-use and extend it. As such, we identified documents to consider for inclusion in three main categories:
Documents included in two existing rapid reviews commissioned by
PHE, as well as additional studies identified using the same search strategies and included in
PHE’s published quarterly literature reviews. These were empirical (quantitative and qualitative) studies of the
NHSHC. The eligibility criteria that were employed in the rapid reviews (and so determined our inclusion of documents from these sources) are summarised in .
Documents retrieved via additional focused searches that we ran to identify additional material that may have been excluded from the existing reviews and bibliographies. This included, for example, relevant commentary or opinion, which may still be included in a realist review where data from these documents may contribute to theory-building.
45 Our more specific search strategies focused on identifying documents focused on the
NHSHC in England. We ran these more targeted searches in MEDLINE, Embase,
CINAHL,
HMIC and Web of Science (SCI-EXPANDED, SSCI indexes). These searches used specific free-text terms describing the
NHSHC alongside relevant subject heading terms as appropriate. The full search strategies used are reproduced in
Appendix 2.
Documents retrieved via a trawl of the NHSHCs website (
www.healthcheck.nhs.uk/), including case studies, local evaluations and abstracts and posters presented at the Health Checks/Cardiovascular Disease Prevention annual conferences from 2014 to 2020. These sources represent an important source of data on local implementations of
NHSHC and additional research studies that have been excluded from previous reviews. Our realist approach provides the opportunity to supplement and structure the informal collaborative knowledge-sharing that has been facilitated by these spaces.
46
Summary of inclusion criteria for PHE-commissioned rapid reviews
This main phase of searching and gathering documents was undertaken in October 2020. At the same time, a regular email alert for (NHS (‘health check’ or ‘health checks’)) was set up using Google Scholar, to help capture research studies and grey literature published over the course of the review. Some additional documents were provided by our professional stakeholder group members and by respondents to our survey, in response to our request for local evaluations or similar documents, as described above.
All documents identified via these processes were stored and deduplicated using Endnote X9 (Clarivate Analytics, Philadelphia, PA, USA) reference management software.
Additional searching
Additional searches for data may be undertaken in a realist review, for example, to help to develop and test particular aspects of the programme theory. Although we had anticipated that additional searches for documents containing empirical data may have been required for this review, the project team agreed that the material had been identified via the searching processes described above were sufficient to meet the needs of this project.
We conducted a small number of focused searches (in Google Scholar) to identify material related to one substantive theory, ‘street-level bureaucracy’,47 which has been used to help to frame and illuminate our final programme theory and discussion, and is described in more detail below (see Chapters 3 and 4). These searches are also reproduced in Appendix 2.
Step 3: Article selection
Documents were exported from Endnote X9 and imported into Rayyan QCRI (Qatar Computing Research Institute (Data Analytics), Doha, Qatar), a web-based tool designed to support screening for systematic reviews. Initially, CD and GW screened a small sample of documents (n = 25) in a pilot process to check for consistency in the application of our initial inclusion and exclusion criteria.
We then screened the full set of documents retrieved in Step 2 using a three-step process. CD screened the titles and abstracts (where available) of the documents retrieved via our searches and the NHSHC website following the eligibility criteria specified in our review protocol and outlined in . GW screened 10% of each of these sets of documents in duplicate (n = 67 documents retrieved via searching; n = 25 documents retrieved from the NHSHC website), and CD and GW discussed discrepancies in decision-making as a means of ensuring consistency in how the inclusion criteria were applied.
Inclusion and exclusion criteria
When this initial screening process was complete, CD re-screened all remaining documents in light of the review’s chosen focus, seeking to identify documents that were likely to include data on what happens after the measurements and risk assessments are complete in an NHSHC. This additional inclusion criterion was applied to identify documents that described, for example, the offer or delivery of advice, signposting or referral, ongoing support or prescriptions following an NHSHC. Where the coverage of any particular document was unclear from the title or abstract, it was included for full text screening. The application of this additional inclusion criterion permitted us to efficiently prioritise documents that were likely to contribute relevant data on our chosen focus.
For the documents included in the existing PHE-commissioned rapid reviews, we relied on the screening processes undertaken by these review teams. We therefore included for consideration in full text all of the documents included in these reviews that related to review questions which focused on the appropriate stage of the NHSHC, as follows:
In the final stage of screening, CD read all documents in full text, to assess whether or not they contained relevant data that could contribute to the development and refinement of the programme theory. This process continued and was repeated during Steps 4 and 5 (see below), as the documents were read and re-read closely multiple times and the analysis evolved over the course of the review.
All excluded documents were stored in an Endnote reference library so that they could be consulted later in the review as required.
At all stages, wherever we identified documents that included relevant data, we also considered the rigour or trustworthiness of those data at the point of inclusion. We did not apply a standard checklist to assess each study. Instead, each piece of extracted relevant data was first assessed in relation to the methods that were used to produce it (where applicable). We did not automatically exclude data that were judged to be of limited rigour, or data that were not produced directly by a specific research method, as we also made an overall assessment of rigour at the level of the programme theory developed over the course of the review.45 Our overall assessment of rigour took into account the role that each piece of data played in the developing programme theory and focused on our judgement of the explanatory value of the theory produced. To make this assessment, we considered both the volume and the nature of the data that underpinned each part of our developing programme theory. We also assessed the plausibility and coherence of each aspect of the programme theory (each context-mechanism-outcome configuration (CMOC)), as well as the relationships between these and the programme theory as a whole. To do so, we applied three interrelated criteria to each of these levels45 (see Step 5 below for more detail):
Consilience: We considered the extent to which the theories explained the included data, and considered theories that explained more of the data to be more plausible than those that could only account for some data (while bearing in mind that the aim in realist research is to identify and explain ‘demi-regular’ patterns of outcomes, i.e. we anticipated some data may point to ‘exceptions to the rule’).
Simplicity: We considered the extent to which the theories were simple and did not require special assumptions to be added to ‘help’ them explain the data.
Analogy: We considered the extent to which the theories fit with what is already known from existing research and substantive (formal) theory and the extent to which the component parts of our theories fit with each other.
All of the review findings presented below were judged by our project team to meet these criteria and so represent our assessment of plausible and coherent theories that explain important outcomes related to what happens after the measurements and risk assessment are completed in an NHSHC. Some aspects of the programme theory are stronger and more plausible than others, being based on a greater volume of more trustworthy data. To ensure transparency, the findings are accompanied by a full account of the data underpinning them: see Table 21, Table 22 and Table 23. A separate data appendix with full details of all extracted data is available on request from the authors.
Step 4: Extracting and organising data
We extracted the main characteristics (bibliographic details and information relating to study design, participants, settings and main findings) of each included document into an Excel spreadsheet. These details are presented in Table 20 in Chapter 3.
The full text of included documents was uploaded into the qualitative data analysis software, NVivo (Version 12, QSR International, Warrington, UK). CD coded relevant sections of text in these documents where they were interpreted as being relevant to what happens after the measurements and risk assessment are completed in an NHSHC. Some coding was deductive (some codes were anticipated in advance of data extraction and analysis, based on the IPT and background reading) but most coding was inductive (codes were created to categorise data in the included studies during the data extraction process) and some was retroductive (codes were created based on an interpretation of the data extracted, where we inferred what the causal force was that generated observed outcomes, i.e. mechanisms). Each new element of data was incorporated into our analysis (as described below in Step 5) and as the analysis progressed and the programme theory was refined over the course of the review, documents were re-scrutinised to ensure that all relevant data were captured. The final version of the coding frame is reproduced in Appendix 3.
As with screening, a 10% set of documents were coded in duplicate. CD and EG coded 10% of documents (n = 22) independently and GW provided an additional check for this coding by reviewing a merged NVivo file. The coding decisions and coding frame were discussed by the wider project team to resolve discrepancies and ensure consistency in how codes were understood and applied. Following this, CD coded the remaining documents following the processes described above. This process is a slight deviation from the process planned in our protocol, which indicated that 10% checks would be conducted by GW alone. This change permitted an additional member of the research team (EG) to provide an additional independent check on data coding, while increasing her familiarity with the data included in the review.
Step 5: Synthesising the evidence and drawing conclusions
We used a realist logic of analysis to make sense of the data included in the review. This process began and moved in parallel with the decisions taken in Step 4, as data were included wherever they were considered sufficiently rigorous and were understood to have relevance to our focus in the NHSHC. When coding of the included documents was complete, the data contained within individual or across closely related categories were read and considered together. We interpreted the extracted data as relating to important contexts, mechanisms or outcomes (or the relationships between these) and used them to build CMOCs, describing why (by which mechanisms) particular outcomes were generated in particular contexts.
To do this, we compiled and interpreted data both within and across included documents. We used cross-case comparison to draw parallels wherever the data demonstrated that similar contexts and mechanisms were in operation to produce patterns of outcomes, and to understand when and how different outcomes were produced. Based on our interpretations, we constructed theoretical explanations in the form of CMOCs for the range of outcomes we observed in the data, where these were relevant to our focus on what happens after the measurements and risk assessment are completed in an NHSHC. We aimed to develop the CMOCs at an appropriate level of abstraction, such that they embodied potentially transferable explanations that encompassed a range of specific circumstances and outcomes. In practice, this meant that our interpretation focused on identifying the salient features of the circumstances described in the data that could be understood as functioning as context, and identifying the specific, proximal outcomes that were produced in those contexts, rather than focusing solely on the intended overall outcomes of the NHSHC. As our theories were refined, more specific CMOCs were merged, whenever they were understood to articulate specific cases of a more abstract phenomenon, while others were separated, when it became clear that more than one active context, mechanism or resulting outcome was in operation. Where mechanisms were not explicitly articulated in the included data, we used retroductive reasoning to infer likely causal forces, with a focus on the reasoning and responses of different participants involved in the NHSHC programme (commissioners, providers and attendees).
Our application of this process to an example CMOC is set out below, to help to illustrate what we did in more detail.
CMOC C1: When commissioners view the NHSHC programme as a means to improve people’s lives through behaviour change (C) they will try to exert their influence over providers to ensure that the programme is delivered with this in mind (O) because they believe this will maximise the potential benefits of the programme (M).
This CMOC is underpinned by data extracted from 12 documents: two research articles (one cohort study and one qualitative interview study), four local evaluations of the NHSHC, three conference abstracts or presentations, one unpublished working document provided by a LA who responded to our survey and two other reports focused on LA roles in programme delivery. The specific data underpinning this CMOC included:
Statements from the commissioners’ perspective (from local evaluations and reports focused on
LA roles, which include local case studies) that illustrate the role of engagement and close working relationships with providers to ensure that NHSHCs are delivered as envisioned. For example:
There should be a dedicated programme team … to address, for example, day-to-day running, timely monitoring, provider quality control (especially training and audit), marketing and programme evaluation.48
Data presented in this report reflect a high-performing [local] authority that works closely with general practices, stakeholders and patient representatives within it.49
The commissioning body … has enthusiastically taken up the findings from their evaluation and created an information package for the participating GP Practices and has also held an information day…50
[The] importance of communications and relationship building should not be underestimated … it is important to build effective, collaborative working relationships between Commissioners and Providers.51
Further statements from the same and additional sources draw attention to the potential for
LA commissioners to use their influence and relationships to work to ensure providers retain a focus on the behaviour change purpose of the
NHSHC. For example:
Liaise with NHS England [Local Area Team] to ensure the information gained from an NHS Health Check is used by … General Practitioners to improve the health of the patient by ensuring there is appropriate: incorporation of the NHS Health Check results into patient records; follow up with their GP; referral to lifestyle interventions as required.48
The results we are most proud of relate to bringing all our primary care practices together to deliver consistency of approach, re-invigorating staff to enable them to feel they can assist people to make the changes.52
‘Softer’ measures of success are seen as: ensuring there is provision of appropriate follow-up lifestyle services to help those with health issues identified during the NHS Health Check. Many commissioners believe this is imperative to the spirit of the programme, even though there are no targets around this.51
(All our emphasis.)
Taken together, our data represent multiple local examples where there was recognition of the importance of a route for LA commissioners to exert influence over NHSHC providers in relation to programme delivery, and that this influence may be used in particular to try to ensure that advice and referrals relating to the programme’s aim to support behaviour change are delivered. By including these observations in local evaluation reports, conference presentations and case studies, commissioners and evaluators have drawn attention to the importance of this role for LAs, and, by implication, a potential disconnection in commissioner and provider priorities that could affect programme delivery. This latter implication is borne out in other CMOCs developed from the commissioner (e.g. CMOCs C2–C10) and the provider perspective (e.g. CMOCs P1–P8), lending further support to CMOC C1 as part of our overall understanding of how the programme operates, as captured in our final programme theory.
Only six of the documents reported research or evaluation work that was undertaken using specific methods, and therefore an overall judgement of the quality and strength of the evidence collated here was required. Our judgement centred on the criteria outlined above in Step 4. We considered that, taken together, our data were consilient, in that CMOC C1 captures a causal explanation for the observed outcome that LA commissioners may work to exert influence over NHSHC programme delivery; simple, in that we did not feel there was need to specify any additional assumptions to support our explanation; and supported by analogy.45 In this case, we did not draw on substantive theory during the development of this CMOC, but we draw attention to our observation that this CMOC makes sense when considered in relation to other CMOCs developed over the course of the review, as summarised in our final programme theory (see Chapter 4).
CD undertook this step and developed and shared sets of developing CMOCs, accompanied by explanatory narratives and their underpinning data, with the wider project team. Following discussion within the team, the CMOCs were refined and re-organised to help develop their explanatory value. This included re-ordering and grouping similar CMOCs together – considering the relationship of each CMOC to a developing overall programme theory – as well as consideration of the level of abstraction at which CMOCs were presented and discussion of potential mechanisms where these were unclear. Emerging findings were also shared with our PPI and professional stakeholder groups (see section Stakeholder groups and above) and feedback from these discussions also informed the project team’s discussions and refinement of the CMOCs.
In the later stages of the review, we also considered whether substantive theory could play a role in supporting or developing our analysis. A small number of documents included in the review included theoretical perspectives, reflecting what we found during IPT development (see Appendix 1), but most of the literature that contributed data to our review was atheoretical. We considered whether substantive theory from various disciplines could help to support or illuminate aspects of our findings during project team discussions.
Overall, analysis and synthesis of the data was an interpretive and iterative process, involving returning to the documents and data and interpreting and re-interpreting their meaning, seeking additional data as needed and ongoing discussion within the project team throughout.
