Stage 4 of the project involved developing the guidance table and explanatory notes, developing the training materials and organising the dissemination of the guidance.
We followed an iterative process to develop the final guidance table and explanatory notes.39–42 An overview of the process followed to write up the guidance table is given in . Although Moher et al.37 provide a brief overview of the guidance development process post consensus study, there is a dearth of literature describing the detail of developing usable guidance from Delphi items. This process was particularly important in eMERGe, because so few items did not reach consensus in the Delphi studies and we realised that we had too many items to form usable guidance in their eDelphi format. Therefore, we decided, as a team, to provide a detailed account here of the process we followed to develop the guidance table and explanatory notes from the Delphi items.
Guidance development process post Delphi.
Project Advisory Group meeting structure
A Project Advisory Group meeting was held in November 2016. A total of 27 people attended the meeting: nine members of the eMERGe project team, one external chairperson for the meeting, seven lay advisors and 10 academic/expert advisors. The purpose of the meeting was to update the Project Advisory Group on the project’s progress, including the results of the audit and eDelphi studies, and to gain its opinion and feedback on the structure of the guidance and next steps in guidance development.
Following presentations on the audit and Delphi studies, Project Advisory Group members discussed and agreed the following points:
Given the initial results of the audit, there was a strong feeling that meta-ethnography authors, peer reviewers and journal editors would all benefit from guidance on meta-ethnography reporting.
When developing the guidance table, there is a need to be pragmatic and realistic about the number of items that can be put into a guidance table for reporting. There was general agreement that, if the guidance included too many items, it was unlikely that they would be used.
The guidance table should be written in such a way that the high-level guidance is relevant to a number of types of user (e.g. a meta-ethnography author, a peer reviewer or an editor of a journal) and is also relevant across clinical and social science disciplines.
The guidance table should reflect what is key to good reporting, with suggestions of how this can be achieved described in the explanatory notes.
Some Delphi items were ambiguous and would need to be reworded, split or merged.
Workshop sessions were held to discuss the structure of the reporting guidance. Project Advisory Group members discussed and gave feedback on the following questions:
How do we group Delphi items into meaningful categories in reporting guidance?
There was general agreement that it would be useful to structure the reporting guidance around the phases of meta-ethnography. Project Advisory Group members felt that structuring the guidance by phases honours the tradition of meta-ethnography and does not force it into another paradigm (e.g. that of systematic reviews or journal article structure). It was suggested that providing the guidance structured into the phases of meta-ethnography could be a useful teaching aid for the conduct and reporting of meta-ethnography, and could minimise the risk of poor reporting of later phases of meta-ethnography, which is especially problematic. The Project Advisory Group suggested that journal formatting could be provided as subheadings within each phase.
Context was discussed as being relevant to several different phases of meta-ethnography, for example the appropriateness of a sample in the context of the research questions. The context of included studies was considered when conducting the translation and synthesis.
What is the minimum requirement to be classified as a meta-ethnography?
There was consensus in the group that meta-ethnography is both a product and a process. Workshop attendees felt that the guidance should reflect underlying principles of meta-ethnography as a cyclical iterative process. The Project Advisory Group agreed that the underlying principle of whether or not a study is a meta-ethnography is whether the seven stages of meta-ethnography are recognisable in the reporting.
There was general agreement that meta-ethnography needs to produce something new, something beyond that which was there before, from the synthesis of primary qualitative data. Suggestions for ‘something new’ included third-order constructs, LOA, new interpretation or a new model. There was agreement that ‘new’ should not be narrowly interpreted, and that further work is needed to define ‘new’ in the context of the product of meta-ethnography.
Do we need essential and desirable reporting items, for example in phase 6 (synthesising translation), and what would they be?
No agreement was reached about whether or not there should be essential and desirable reporting items in the guidance. Some attendees felt that the guidance should clarify what must be included in meta-ethnography reporting, and what would be useful to include if there is space in the paper. Others felt that the guidance should contain overarching principles for meta-ethnography reporting, with more detailed notes on what authors might do to meet these principles.
Do we need separate meta-ethnography reporting guidance or an extension to existing reporting guidelines (and, if so, to which guideline)?
There was agreement among workshop attendees that the guidance should stand alone, rather than as an extension to existing guidelines for qualitative synthesis. It was felt that there may be confusion for users as to where the meta-ethnography guidance maps to existing standards, if the guidance was developed as an extension to, for example ENTREQ
34 or PRISMA.
32What should be included in the explanatory notes?
There was discussion about the use of exemplars to illustrate good reporting practice. Some attendees felt that exemplars could be valuable; however, others were concerned that exemplars would become ‘the new set of words people use’ and may endorse one particular way of doing things over other ways. The Project Advisory Group suggested providing exemplars as part of the training resources on the website, rather than in the reporting guidance paper.
Following these extensive discussions at the Project Advisory Group meeting, the project team agreed that:
The guidance would be standalone rather than an extension to existing guidelines.
There would be two levels of guidance – (1) essential and (2) desirable.
The guidance would be structured in eight phases: phase 0, which was introduced by the project team following stage 1 of the eMERGe project, and Noblit and Hare’s
25 seven phases of meta-ethnography.
A study is not a meta-ethnography unless the specific phases are followed. The study must also use qualitative primary data. It must also come up with something new, although the definition of what constitutes something new is very broad.
We would hold two online conferencing sessions with the Project Advisory Group to seek feedback on further iterations of the draft guidance. These had not originally been planned in the study protocol; however, we decided to introduce these sessions to give us further essential feedback on the guidance.
As a result of the discussion and decisions made at the meeting and workshops, the items were restructured into the seven phases of meta-ethnography plus phase 0. A small writing group was formed as a subgroup of the main project team, consisting of five members (MC, EASD, NR, IU and RJR). This writing group discussed each of the items and agreed whether they were essential or desirable for meta-ethnography reporting. Essential items were highlighted in the guidance document. No other changes were made to the wording or content of the items at this time.
Project team meeting: merging items
The project team held a meeting in January 2017 to review the new structure of the guidance, agree the selection of essential items and discuss which items could be merged to reduce the reporting criteria to a manageable number. The project team agreed that items that were related should be merged to form larger items. The project team also agreed that merging items did not mean losing any content, just avoiding repeats and cutting wording.
Items that had been included in the new phase 0 were merged into items in phase 1. As the rationale for using meta-ethnography instead of other qualitative evidence synthesis methods was now contained in phase 1, it was decided that the guidance should not have a phase 0 and should revert back to Noblit and Hare’s25 original seven phases of meta-ethnography. Phase 1 was renamed ‘Selecting meta-ethnography and getting started’, to reflect the importance of reporting why meta-ethnography was chosen as the appropriate qualitative evidence synthesis approach.
The team discussed the role of context in the guidance. The team agreed that there was a need to be explicit on what is meant by context in a meta-ethnography. The critical importance of context in meta-ethnography was identified across all stages of the study (e.g. the audit results indicated that context of studies is not well reported in meta-ethnographies at present). Several contextual factors should be considered when reporting a meta-ethnography, including the context of the review question and the context of primary studies. Context is essential in order to interpret the meta-ethnography for use in policy and practice.
As a result of the decisions made at the project team meeting, the guidance was structured into seven phases and the number of reporting items was reduced from 49 to 27.
Project Advisory Group online sessions: usability
We held two online conferencing sessions with Project Advisory Group members in February 2017. Prior to the sessions, all members were sent copies of the guidance, with the 27 items structured in the seven phases. Six members attended the online sessions and a further three members sent written feedback on the draft guidance. Feedback from the Project Advisory Group members primarily focused on the usability of the guidance, with key points including:
consistent level of detail in the guidance table, with further detail to be supplied in the explanatory notes
importance of explaining how context should be considered in different phases in the guidance
make changes to item ordering so that guidance follows a logical progression (while recognising that meta-ethnography is an iterative process)
present the guidance in such a way that there are clear criteria covering the process to be reported, without being too prescriptive about how these are reported
reduce number of items
increase clarity for some of the items – what exactly does the reviewer need to report?
importance of highlighting iterative nature of meta-ethnography
abstract does not fit within the meta-ethnography phases and should, therefore, stand alone, not as part of guidance table.
The members who attended the online sessions also discussed whether or not it was appropriate to highlight essential and desirable items. The group agreed that this was less of an issue now that there were fewer items, and that the focus should be on reporting information for all the criteria in the guidance table.
There was some discussion about the use of exemplars in the explanatory notes. We had originally planned to include examples of good reporting from published articles in the explanatory notes. However, concern was raised during the sessions that providing exemplars would lead to verbatim reporting in a particular way, rather than encouraging creativity in how reviewers report meta-ethnography findings. There was a strong feeling that providing exemplars may be too prescriptive.
Writing group: guidance refinement
Following the online sessions, two members of the project writing group (MC and NR) restructured the guidance table, bearing in mind all the feedback from the Project Advisory Group. We merged further items and then carefully extracted the content into two levels of reporting: (1) a high-level summary of the reporting criteria for the guidance table and the detailed explanatory notes that provided additional clarification and (2) guidance that could not be provided in the summary guidance table. We structured the items within the phases under subheadings based on the journal article section the information would best fit into. In particular, we drew a distinction between reporting process and results of the different phases of meta-ethnography: clarifying what information should be provided in each phase about what was done (methods) and what was found (results). As a result of this restructuring, the number of items in the guidance reduced from 27 to 21.
A third member of the writing group (IU) then checked the reporting criteria and explanatory notes against the items that had reached consensus in the Delphi studies to (1) check that no item had been missed from the rewriting process and (2) identify if any further detail had been added to the guidance. Extra detail had been added to the explanatory notes in 16 places and in each case the writing team identified where the additional information had come from (e.g. item 10, phase 3, ‘Describe characteristics of the included studies’) – further information was added to the explanatory notes from the online conferencing sessions about clarifying the availability of contextual information in the primary studies: ‘If such contextual information is not available in the original papers, review authors should make this clear in their report to readers (e.g. as footnote).’ Additional detail had been added to the explanatory notes from the Project Advisory Group meeting, the project team meeting [including discussion with Professor George Noblit during his visit to the eMERGe project (June 2016)], online discussions and from the audit findings.
Following this, we sent a copy of the draft guidance to the seven Project Advisory Group lay members, specifically asking for their feedback on readability, clarity and asking them to identify words they thought that we should define in a glossary on the website. This step was carried out to ensure that the guidance and explanatory notes would be understandable and usable for a wide audience, and to supplement the training material to be provided on the project website. Five lay members responded with comments and glossary suggestions. The feedback from lay members was very positive and they all commented on the clarity and readability of the guidance:
. . . I was delighted to find how well you had educated us during this process and felt overall that if I were a young researcher in the field, approaching a new piece of work, I would have been able to find structure and clarity in these guidelines . . .
Project Advisory Group lay member, March 2017
Project team meeting: wording, style and extensions
The project team held a meeting in March 2017 to review the draft guidance. The project team decided to create three extensions to the guidance for reporting steps and processes that are not common to every meta-ethnography. The three extensions cover (1) format and content of the meta-ethnography outputs (e.g. title, abstract and keywords), (2) assessment of methodological strengths and limitations of included primary studies (e.g. quality appraisal) and (3) assessment of confidence in synthesised qualitative findings using Grading of Recommendations Assessment, Development and Evaluation (GRADE) CERQual.14,38
The team agreed consistent wording for the guidance (e.g. that the person doing a meta-ethnography should be called a ‘reviewer’ in the guidance).
The team agreed to merge the items on reflexivity to create one item that considered the internal and external context and methodological aspects of the synthesis.
There was discussion about the use of exemplars in the explanatory notes – we had received mixed feedback during the Project Advisory Group meeting and online sessions about the use of exemplars. Although exemplars can help to illustrate a point, there was feeling among some members of the Project Advisory Group and members of the project team that there are a number of ways the guidance can be met, and we did not want to be too prescriptive about how people provide information, so long as they do provide the content to meet the guidance. It was felt that providing exemplars in the explanatory notes to the guidance table may inevitably lead to new meta-ethnography reviewers copying existing formats for reporting, rather than developing their own creative ways to meet the reporting criteria. The project team noted that a further issue with exemplars had been identified during the audit. It was clear from the audit results that, although one paper may be an exemplar for reporting a particular phase of meta-ethnography, it may not be a good exemplar for reporting the other phases. The project team was concerned that if we highlighted a paper as a specific exemplar of good reporting for one phase, over time, this distinction could be lost and the paper could be considered an overall example of good reporting when this was not the case. The team agreed to follow the Project Advisory Group’s previous suggestion and place exemplars on the project website as part of the training materials, rather than including exemplars in the reporting guidance document.
Guidance writing: final processes
One team member (MC) removed detail from the guidance table and explanatory notes that related to the new extensions and merged the reflexivity items. Therefore, the number of items in the final guidance reduced from 21 to 19.
Changes to style and wording were made in line with the project team meeting decisions.
One member of the project team (IU) generated a list of notes from stage 1 (systematic review) and stage 2 (audit) findings relevant to each of the final 19 guidance items. Another team member (MC) checked the explanatory notes for each item against the notes from stage 1 and stage 2 and added detail, where appropriate, about the justification for each item from the literature. A final check was then conducted by one researcher (IU) of the detailed explanatory notes against the stage 3 Delphi items that met consensus, to ensure that the meaning of each item retained fidelity to the Delphi items. We felt that conducting these checks against each of the previous stages of the project was important to (1) ensure that we remained faithful to the consensus achieved in the eDelphi studies and (2) reduce the risk of bias acquired through being immersed in meta-ethnography reporting over a 2-year period and, as a result, developing our own expertise and opinions about meta-ethnography.
The first two extensions, (1) ‘Format and content of the meta-ethnography outputs’ and (2) ‘Assessment of methodological strengths and limitations of included primary studies’, were written from the points removed from the guidance table and explanatory notes. The third extension, ‘Assessment of confidence in synthesised qualitative findings using GRADE CERQual’, was written by a member of the project team (JN) who was involved in developing CERQual in collaboration with the other CERQual originators. This extension was considered essential because we wanted to link the eMERGe reporting guidance with other developments in the field.
The final guidance table, explanatory notes and extensions were sent out to the project team and Project Advisory Group members who qualified for authorship for final feedback.
Two significant changes were made to the guidance tables and explanatory notes as a result of feedback:
It was felt that the explanatory notes for reporting criterion 6, phase 2, ‘Searching processes – Describe how the literature searching was carried out and by whom’, were not sufficiently comprehensive. The process for identifying meta-ethnography-specific reporting principles had not been designed to generate guidance on the detail of conducting a literature search. A decision was made by the project team to cross-reference to existing published guidance on searching for qualitative evidence, recommending that reviewers ‘follow an appropriate guideline for reporting qualitative literature searches (e.g. STARLITE’
89).
It was felt that reporting criterion 18, phase 7, which had been named ‘Reflexivity’, also covered the strengths and limitations of the meta-ethnography process. The project team decided to rename reporting criterion 18, ‘Strengths, Limitations and Reflexivity’, to better represent the type of issues that needed to be considered by those writing meta-ethnography reports.
The final guidance table is provided in Appendix 9. We also developed detailed explanatory notes and extensions to accompany the guidance table; these are available in the full published reporting guidance as an open access journal article.39–42
Training materials
The project team decided to create a range of online training materials to support the project output, hosted on the project website. We decided to produce online material rather than a one-off real-world seminar, as this offered greater potential for dissemination and online content would be more accessible to users. We produced a range of training materials to ensure that the material was useful to a wider range of viewers, for example students, lay people, end-users and academics. The training materials include:
a glossary of terms, defining specialist words identified by Project Advisory Group lay members
exemplars for each of the reporting criteria in the guidance table
four films following a junior researcher on her journey to understand more about meta-ethnography and reporting meta-ethnography –
– meta-ethnography then and now with Professor George Noblit
– the eMERGe project – development of the reporting guidance with Dr Emma France
– the eMERGe reporting guidance – the wider policy and practice context with Professor Jane Noyes
– the eMERGe reporting guidance – format, content and use with Dr Nicola Ring.
The project team held a webinar in May 2017: ‘Introducing the New Meta-ethnography Reporting Guidance – what it is and how to use it’. This free, 1-hour webinar gave an overview of why the reporting guidance is needed, what format the guidance takes and how to use the guidance, and gave attendees the opportunity to ask questions. Fifty people from around the world attended the webinar. Attendees included PhD students and academics. A full list of the training resources developed by the project team is available in Appendix 10. A recording of the webinar and copy of the associated slides is also available on the eMERGe project website (www.emergeproject.org/resources; accessed 26 March 2018).
