From Shepherd et al.55 and Kavanagh et al.70
1. Screening
1A. Is this a Marmot topic?
Guidance: Marmot topics:
health risks (e.g. smoking, obesity, drug/alcohol use, healthy eating, physical activity)
mortality risks (e.g. cardiovascular disease, cancer prevention)
best start in life (e.g. antenatal care, parenting, immunisation, teenage pregnancy)
employee health/safety (e.g. injury prevention)
illness prevention (general health promotion)
If no, stop coding.
1B. Is this a process evaluation?
If no, stop coding.
2. Which processes were evaluated?
Guidance: Tick as many as appropriate. Specify further where possible
Perceptions, understanding or acceptability of an intervention
Accessibility of the intervention/programme reach
Consultation or collaboration
Content of the intervention
Implementation of the intervention
Costs associated with the intervention
Management and responsibility
Quality of the programme materials
Skills and training of the intervention providers
Other (please add details)
3. About which processes do authors offer conclusions?
Guidance: Tick as many as appropriate. Specify further where possible
Perceptions, understanding or acceptability of an intervention
Accessibility of the intervention/programme reach
Consultation or collaboration
Content of the intervention
Implementation of the intervention
Costs associated with the intervention
Management and responsibility
Quality of the programme materials
Skills and training of the intervention providers
Other (please add details)
4. What methods were used to collect data on the processes involved?
Guidance: Tick as many as appropriate. Specify further where possible
5. Who were data collected from?
6. When did the evaluation take place in relation to the intervention?
Tick as many as appropriate. Specify further where possible
7. Were steps taken to increase rigour/minimise bias and error in the sampling for the process evaluation?
Guidance: Consider whether:
the sampling strategy was appropriate to the questions posed in the process evaluation (e.g. was the strategy well reasoned and justified?)
attempts were made to include all relevant stakeholders and/or obtain a diverse sample (think about who might have been excluded who may have had a different perspective to offer)
characteristics of the sample critical to the understanding of the study context and findings were presented (i.e. do we know who the participants are in terms of, for example, role in the intervention/evaluation, basic sociodemographics, etc.)
Yes, fairly thorough attempt made – please specify
Yes, several steps taken – please specify
Yes, a few steps taken – please specify
Unclear/can’t tell/not stated – please specify
No, not at all
8. Were steps taken to increase rigour/minimise bias and error in the data collected for the process evaluation?
Guidance: Consider whether:
data collection tools were piloted/validated (if quantitative)
data collection was comprehensive, flexible and/or sensitive enough to provide a complete and/or vivid and rich description/evaluation of the processes involved in the intervention [e.g. Did the researchers spend sufficient time at the site/with participants? Did they keep ‘following up’? Were steps taken to ensure that all participants were able and willing to contribute? (e.g. confidentiality, language barriers, power relations between adults and young people) Was more than one method of data collection used? Was there a balance between closed and open-ended data collection methods?]
Yes, fairly thorough attempt made – please specify
Yes, several steps taken – please specify
Yes, a few steps taken – please specify
Unclear/can’t tell/not stated – please specify
No, not at all – please specify
9. Were steps taken to increase rigour/minimise bias and error in the analysis of the process data?
Guidance: Consider whether:
data analysis methods were systematic (e.g. was a method described/can a method be discerned?)
diversity in perspective was explored
the analysis was balanced in the extent to which it was guided by preconceptions or by the data (i.e. participants’ views, researcher observations, etc.)
the analysis sought to rule out alternative explanations for findings (in qualitative research this could be done by, for example, searching for negative cases/exceptions, feeding back preliminary results to participants, asking a colleague to review the data, or reflexivity; in quantitative research this may be done by, for example, significance testing)
Yes, fairly thorough attempt made – please specify
Yes, several steps taken – please specify
Yes, a few steps taken – please specify
Unclear/can’t tell/Not stated – please specify
No, not at all – please specify
10. Were the findings of the process evaluation grounded in/supported by the data?
Guidance: Consider whether:
enough data are presented to show how the authors arrived at their findings
the data presented fit the interpretation/support claims about patterns in data
the data presented illuminate/illustrate the findings
(for qualitative studies) quotes are numbered or otherwise identified so that the reader can see that they don’t just come from one or two people
Reasonably well grounded/supported – please specify
Fairly well grounded/supported – please specify
Limited grounding/support – please specify
11. Please rate the findings of the process evaluation in terms of their breadth and depth
Guidance: Consider whether:
(NB: it may be helpful to consider ‘breadth’ as the extent of description and ‘depth’ as the extent to which data have been transformed/analysed)
a range of processes/issues were covered in the evaluation
the perspectives of participants are fully explored in terms of breadth (contrast of two or more perspectives) and depth (insight into a single perspective)
both the strengths and weaknesses of the intervention are described/explored
the context of the intervention has been fully described/explored
richness and complexity has been portrayed (e.g. variation explained, meanings illuminated)
there has been theoretical/conceptual development
Limited breadth or depth
Good/fair breadth but very little depth
Good/fair depth but very little breadth
Good/fair breadth and depth
12. To what extent does the process evaluation privilege the perspectives and experiences of the public?
Guidance: Consider whether:
the public are included in the process evaluation
there was a balance between open-ended and fixed-response options
whether the community were involved in designing the research
there was a balance between the use of an a priori coding framework and induction in the analysis
the position of the researchers (did they consider it important to listen to the perspectives of the community?)
steps were taken to assure confidentiality and put people at their ease
Not at all – please specify
A little – please specify
Somewhat – please specify
A lot – please specify
13. Overall, what weight would you assign to this process evaluation in terms of the reliability of its findings?
Guidance: Think (mainly) about the answers you have given to questions 7–10 above.
Low reliability
Medium reliability
High reliability
14. What weight would you assign to this process evaluation in terms of the usefulness of its findings?
Guidance: Think (mainly) about the answers you have given to questions 10–12 above and consider:
how well intervention processes are described (e.g. does it provide useful information on barriers and facilitators to implementation – factors that others implementing the intervention would need to consider?)
whether the findings can help us to explain the relationship between intervention process and outcome (e.g. why the intervention worked or did not work; factors influencing effectiveness; how the intervention achieved its effects)
Low usefulness
Medium usefulness
High usefulness
