Literature Summary

The terminology surrounding complex and multicomponent interventions can be a major hurdle for systematic reviewers. In the literature, we found the following definitions of “complex” interventions:

• “Complex interventions in health care, whether therapeutic or preventative, comprise a number of separate elements which seem essential to the proper functioning of the intervention although the ‘active ingredient’ of the intervention that is effective is difficult to specify…The greater the difficulty in defining precisely what, exactly, are the ‘active ingredients’ of an intervention and how they relate to each other, the greater the likelihood that you are dealing with a complex intervention.”²
• Craig et al.³ acknowledged that “there is no sharp boundary between simple and complex interventions,” and provide five aspects that point to complexity: (1) number of interacting components within the experimental and control interventions; (2) number and difficulty of behaviors required by those delivering or receiving the intervention; (3) number of groups or organizational levels targeted by the intervention; (4) number and variability of outcomes; and (5) degree of flexibility or tailoring of the intervention permitted.
• Pawson et al.⁴ described seven features of “complex policy interventions”: (1) theory-based; (2) requiring the active input of people; (3) involving a chain of processes for implementation; (4) the chain is nonlinear and may work in reverse; (5) heavily influenced by context; (6) ideas from newly involved people can change the intervention; and (7) feedback from implementation can change the intervention.

For “quality improvement” interventions, a paper by Alexander⁵ defined such interventions as:

actions for improving the processes and outcomes of health care, including increasing value; improving responsiveness to customers and consumers; improving outcomes in the areas of safety, effectiveness, timeliness, patient centeredness, equity, and efficiency; reducing variation in outcomes; and increasing organizational adoption and implementation of continuous improvement methods in ongoing operations.

This far-reaching text captures important concepts, but it is hard to imagine which actions, if any, it excludes.

We view the word “complex” as a general term that can describe both “multicomponent” and “health systems” interventions. It could refer to the intervention itself, the setting in which the intervention is implemented, the numbers of care specialties involved, patient comorbidities, or other facets.

Key Informant Interviews

KIs agreed that the terminology used in this field is inconsistent. They had diverse opinions on the usefulness of the term complex. Some felt complex and multicomponent are used interchangeably, others felt it is critical to distinguish between complex and multicomponent interventions, and others felt that the term complex is not helpful in describing interventions and should not be used. As a group they also felt that almost all interventions could be considered complex in some way, and so the word complex may be unnecessary if it is used only to describe the intervention itself. For example, one said “…even a ‘simple intervention’ is embedded in a complex set of social relationships and workflows. So I don't know that the complexity part of it helps.” Table C-1 in Appendix C presents KIs' thoughts about terminology and definitions relating to multicomponent, complex, multifaceted, and multidimensional interventions.

Without a clear gold standard for terminology, KIs recommended that reviews should clearly define their terms. For example, one said “I think the terminology is a problem in the area and I think anyone really needs to think about what dimensions are important for the work they're doing and really just be clear about what terms you're going to use and what they mean, because there's no shared view of it.”

The word multicomponent, by contrast, was generally recommended by the interviewees. One said, “I think, that the value about distinguishing multicomponent interventions are because there are questions about the interaction of the components, which components are critical and variation across the individual components, across different studies. So thinking about them and how to collect information is useful.”

Some terminology confusion is due to conflating a multicomponent intervention with a multicomponent implementation of an intervention. One expert pointed out that any intervention, no matter how simple or complex, can involve multicomponent implementation, because health care administration has many moving parts. Thus, implementation difficulties may always exist, and are not unique to multicomponent interventions. Terms such as “health systems” may help to address the latter concept. With these challenges in mind, we chose to focus on the unique problems that arise, for a systematic reviewer, when an intervention has multiple components.

Another important issue raised by systematic reviewers, researchers, and implementers is what counts as a component. The answer is often unclear, and different reviewers can reasonably disagree. Some may feel (as did one of our KIs) that a set of educational efforts (e.g., seminars, brochures, and Web sites) are all education and therefore represent a single component, not multiple components. Others, however, felt that if an action can be separated from other actions, and can be reasonably expected to exert an independent influence, then it can be called a component: “I would say it's multicomponent to the extent that you think the individual components have separate effects and may differ within their effects.” Labeling what is and is not a component may not be as helpful as understanding how the parts of an intervention have to fit together to have an effect. In general, the lumping or splitting of components is topic-dependent and is left to the judgment of the systematic reviewer.

One expert noted that for multicomponent interventions, the components can be “fixed” or “variable.” When the components always occur together, they are fixed, and can be treated as a bundle of interventions. Normally, however, the components are variable, meaning some implementations may involve only some components. A further delineation is made between the targets of the intervention: the individual patient, a patient population, or a health system.

Summary

There are very few truly simple interventions. The consensus from the experts we interviewed was that the word complex is unnecessary, due to its lack of specificity. Some suggested that all multicomponent interventions are complex, whereas others felt that “complexity” requires additional things (such as the ones listed in the definition above by Craig, et al.³).

While differing thoughts and opinions exist in the field regarding terminology and definitions for multicomponent and complex interventions, no matter what terminology is used, when describing the interventions themselves, clear definitions are critical.

Literature Summary

A common first step in a review is to construct a framework and Key Questions that clarify the clinical logic, potential linkages, and what information is being sought. Interventions may be defined at varying levels of granularity. Multicomponent interventions with the same general intent can include a range of components and activities. A class of interventions for a topic may include a type of intervention (e.g., lifestyle intervention); may consist of several components (e.g., support and education for nutrition, exercise, and psychosocial issues); and each component may include varying activities (e.g., for nutrition, weekly nutrition classes, a daily food diary, and a food scale).

The framing of a review for complex interventions presents particular challenges. The Cochrane Handbook notes:

“For more complex interventions…the common or core features of the interventions will need to be defined. It is useful to consider exactly what is delivered, at what intensity, how often it is delivered, who delivers it, and whether people involved in delivery of the intervention need to be trained. Review authors should also consider whether variation in the intervention (e.g., based on dosage/intensity, mode of delivery, frequency, duration) is so great that it would have substantially different effects on the participants and outcomes of interest, and hence may be important to restrict.”⁶

In the literature we identified a number of approaches to these many considerations in systematic reviews and guidance documents. Many fit within the commonly-used PICOTS framework (Patient population, Intervention, Comparator, Outcomes, Timing, Setting); they focus on ways of categorizing the intervention or its components, the setting, or both. Other approaches were theoretical in nature, and depart from PICOTS. We further organized these approaches into three broad categories based on the focus or purpose of the systematic review and relation to PICOTS (Figure 2).

Figure 2

Systematic review questions and approaches to conceptualizing interventions.

For questions focusing on the overall effectiveness, systematic reviews may choose to overlook variability in the number, type, and intensity of intervention components. Reviews seeking to understand the comparative effectiveness of interventions or effectiveness of certain subgroups of interventions address the heterogeneity within these multicomponent interventions by categorizing interventions by commonalities. Finally, reviews seeking to answer more complex questions about the reasons for the success or failure of interventions may use a theory-driven approach. As one moves from left to right in Figure 2, the complexity of the review question increases. More complex questions often require more detailed sources of information, which may extend beyond the typical systematic review sources of published and unpublished studies to include qualitative insights from KIs, primary data, and increased application of judgment from the systematic reviewer on the nature and use of these additional data sources.⁷ Table 2 presents a summary of the framework approaches.

Table 2

Summary of framework approaches.

Key Informant Interviews

Table C-2 in Appendix C presents KI's thoughts on frameworks and approaches to systematic reviews of complex multicomponent health care interventions. Overall, systematic reviewers, researchers, and implementers agree that it is important for reviews of multicomponent interventions to have a framework. Frameworks outline the clinical logic and frame the approach for identification, analysis, and reporting. KIs mentioned using or seeing various approaches to framing reviews of complex multicomponent interventions ranging from the commonly used PICOTs framework to the Donabedian framework for quality improvement to realist reviews. One KI specifically mentioned that the realist approach was not helpful: “So, realist synthesis is an example of that, just sort of unhelpful jargon,” however there was general agreement that PICOTS was a reasonable starting point. KIs reported often requiring greater depth on specific areas of the PICOTS for complex multicomponent health care interventions. Some reported that the greater depth was needed to describe the intervention: “We spent a lot of time trying to use it (PICOTS) for our review, and honestly it just fell apart for us…intervention and comparator just doesn't come anywhere close to capturing the complexity.” One KI noted that some groups have proposed variations of PICOTS to more explicitly and consistently describe elements for topic areas: “(PICOTS) is generally serviceable, so long as you put enough effort into…the characterization of the setting.” Some KIs reported needing a way to add the implementation process in PICOTS: “I've had a number of groups who have produced guidelines around the PICOTS sort of framing, and I've never found them very useful. When you're doing process work you frame them at the level of process.” Lastly, some KIs related the importance of grounding reviews based upon a theoretical framing of how the intervention is thought to work: “I would want to have like a logic model of some sort in conjunction with a PICOT to help formulate and flesh out the PICOT statement.” “I think having some sort of theoretical model of how the interventions work may help.” Overall however, there was general agreement that a PICOTS framework could be serviceable if additional augmentations were included in reviews of complex multicomponent health care interventions. KIs mentioned the importance of assembling technical experts, recipients of services, and implementers to flesh out the relevant dimensions of the review's framework.

Summary

A variety of approaches have been used to frame systematic reviews of complex multicomponent health care interventions. There is unlikely to be one accepted standard that will fit all review topics. The PICOTS approach is familiar to EPC reviewers and is widely understood due to its broad usage. The selection of whether to use PICOTS and/or to supplement or replace PICOTS with another framework depends on the topic and whether the intent of the review is to determine overall effectiveness (lumping) or to evaluate the effectiveness of various components. For quality improvement topics, a Donabedian (structure, process, outcome) format may be more easily understood either in place of or in addition to PICOTS. For other topics such as behavioral interventions, a taxonomy or functional approach may be needed in addition to PICOTS to delineate the distinguishing features that determine behavior change. Technical experts, stakeholders, implementers, and clinicians should be engaged early in the review to understand the dimensions that are important to the review and to understand what framing is more easily understood by the users in the field.

Literature Summary

Comprehensive searching is critical to systematic reviews, yet the many dimensions and inconsistent terminology of complex, multicomponent interventions poses substantive challenges. Additionally, because many complex, multicomponent interventions are implemented as part of quality improvement across health care organizations, results are not necessarily published in the traditional literature. Two articles specifically discussed search-related challenges involving complex multicomponent interventions:

• Greenhalgh, et al.³⁴ described search methods for a systematic review of the “diffusion of service-level innovations in health care organisations.” They used three general methods: protocol-driven (i.e., a priori search strategy), snowballing (i.e., articles found based on reference lists or citations of other articles), and personal knowledge (i.e., using the team's expertise, or serendipitous discovery). Interestingly, the snowballing method was the detection method for more articles (51 percent of empirical studies that were later included) than protocol-driven search (40 percent). Further, the review included even more nonempirical articles (e.g., reviews) than empirical studies, and these were also more likely to have been found by snowballing than by a protocol-driven search.
• Shepperd, et al.²³ pointed out that terminology confusion in the field has made literature searching more difficult. They encouraged searchers to use iterative scoping, look for unpublished studies, review reference lists/citations, and consider non-health care journals in other fields such as engineering, social sciences, and management.

Key Informant Interviews

While KIs were not specifically asked about issues in searching for evidence on complex multicomponent interventions, some discussed relevant issues. In particular, they talked about challenges in finding relevant reports, as many implementations would not be published in the traditional literature sources, given their implementation, quality improvement, and health systems nature. One KI spoke about contacting subject matter experts:

We basically identified subject matter experts and then had them link in to contacts at each of the high performing hospital systems, to say what have you done, what is your experience, has this been published, and really codify that information, as well as doing a systematic literature search in each one of those and then synthesizing that information at an expert panel meeting, and that's how we came up with that matrix.

One KI mentioned the importance of extending searches beyond traditional medical search engines due to concerns about publication bias: “the problem is the literature tends to be very biased and there are a lot of fundamental reasons why that's true. So, for example, many journals won't publish negative results …So I think there's a bias in the literature to begin with and so you can't just limit yourself to the peer review literature.”

Summary

Literature Summary

The array of study types used by primary researchers to assess complex interventions is broad. In the Medical Research Council (MRC) updated guidance for the development and evaluation of complex interventions, the authors acknowledge that a randomized controlled trial (RCT) may the optimal study design to minimize bias but researchers may consider other study types depending on the type of question and circumstances. The study types that could be used to evaluate complex interventions include individually randomized trials, cluster randomized trials, stepped wedge, preference trials and randomized consent, and N of 1 designs.³ Ovretveit et al.,³⁵ in guidance for evaluation of complex social interventions, described three main categories of study types for evaluation: experimental and quasi-experimental design; observational, such as cohort evaluation, case evaluation and realist evaluation; and action evaluation, which provides early feedback to implementers for adaptation or change. Primary researchers may have constraints related to resources, policy, and other circumstances which may require trade-offs between the importance of the intervention and the value of the evidence,³ and lead to the use of alternative study designs for evaluation. Primary researchers may choose a nonrandomized study for evaluation because they seek an answer to a different question; they may choose not to “control out” factors but instead better understand their influence or focus on implementation issues. Some seek to answer questions other than, “is it effective?”, and use methods which seek to understand how the intervention causes its effects.³⁵ Choice of study type may depend on feasibility, acceptability, resources, time, type of question asked, and type of intervention to be studied.³⁵

Many sources in our literature review called for the inclusion of a broader array of study types in systematic reviews of complex multicomponent interventions,^{12, 23} citing the limitations of relying solely on RCTs for assessment of complex interventions,^{2, 35, 36} though also noting that non-RCTs may be more prone to bias.³⁷ RCTs may not sufficiently capture contextual factors that may affect the outcome,³⁸ may not reflect the dynamic and interacting nature of multicomponent and systems-level interventions, and need for adaptation during implementation. They may also fail to sufficiently capture intermediate outcomes that may be part of the causal pathway.³⁶ More practically, few RCTs may be found for some topics because they may not be feasible, ethical, or appropriate.

In the CRD manual for systematic review, authors recommend that inclusion criteria be dictated by the types of relevant interventions included in a systematic review and the needs of the end users, rather than by study type.^{12, 21} Restricting a review to RCTs may limit the scope to a smaller set of interventions, and other interventions relevant to end-users may be missed. Nonrandomized studies may make up a larger proportion of research in areas such as behavioral, health systems, and quality improvement interventions.^{5, 39} For example, an examination of EPOC's health systems governance and financial systematic reviews found that the most common group of nonrandomized studies were controlled before-after studies.³⁹

Other study designs can complement RCT evidence by providing evidence about the relative contribution of components of an intervention⁵ or provide useful and reliable information that can help to further explain the results of RCT findings.⁴⁰ Qualitative data may provide a fuller understanding of the interventions included in a systematic review. Other types of documents such as a policy documents can inform reviewers about the rationale for intervention development or constraints.

Qualitative studies performed alongside an RCT can promote an understanding of the intervention for individual studies in the evidence base. They may identify active components of the intervention, provide insight into intervention development, and provide information about implementation fidelity.³⁹ Studies of this type are not common, and usually describe the development of an intervention prior to an RCT. While this may inform reviewers of what the study investigators intended, it may not provide insight into what was actually delivered.³⁹ Lewin, et al.³⁹ found that while qualitative components are increasingly included in RCTs, they are still relatively uncommon. It was often difficult to identify the related qualitative study, and a minority of authors specifically noted the use of a mixed methods approach in their papers.³⁹

Contextual and other implementation factors may be poorly reported in RCTs, and other sources may provide this detail. Qualitative studies, process evaluations, and implementation studies may describe implementation and contextual details and inform questions about adoption and sustainability.⁵ The information may be used to inform factors in meta-regression¹⁰ or explore the heterogeneity of results through analysis of implementation or contextual factors.⁴¹ Victora et al.³⁶ notes that program evaluations can complement RCT evidence by providing information about implementation issues and evidence about intermediate outcomes related to behavior modification necessary for intervention impact, which may be important when the causal pathway between intervention and outcome is long and complex. In addition, they may also explore how behavior affects the intervention dose and dose-response across populations.³⁶

While the inclusion of alternative study types may contribute to or inform the evidence, this may present challenges in practice. Lenz et al.²¹ in their review of systematic reviews of diabetes and hypertension self-management programs, found that most systematic reviews excluded study types other than RCTs and did not capture relevant studies that could contribute to an understanding of intervention development, evaluation, and implementation. They also noted that difficulties in identifying these studies. While restricting inclusion to RCTs may not capture relevant studies that could potentially contribute to the evidence base, Glenton et al. noted that even when nonrandomized trials are considered, the degree to which they are found and poor study quality leads to variability in their influence on review conclusions.³⁷ Other issues similar to those concerning RCTs involving reporting and adequate descriptions of interventions may also limit usefulness. The CRD manual notes that the preferred strategy may be to use the best available evidence, as opposed to the best evidence. They state however that a broader array of study designs may increase the complexity of searching, quality assessment, and synthesis.²⁰ Investigators must consider the advantages and disadvantages of various study types, as well as the value of their relative contribution to the evidence base.

Key Informant Interviews

As shown in Table C-3 in Appendix C, overall, systematic reviewers, researchers, and implementers feel that including a broad range of studies is important in reviews of multicomponent interventions. There was general consensus that the floor for inclusion should not necessarily be based upon particular study designs, but rather other measures of quality and validity. One KI stated:

…my second best (to RCTs) would include a set of at least three types of studies: (1) a nicely done time series, (2) a controlled before and after that the comparison group was reasonably similar and there wasn't a lot of risk of confounds or selection wise problems going on, and (3) …cluster design … I think a traditional before and after, one shot, no control or comparison group would have a hard time convincing me that it was above the floor.

Although not commonly used in systematic reviews, KIs mentioned the importance of qualitative studies stating that they informed “judgments about the applicability of the intervention and the feasibility of the intervention primarily.” “The main role of qualitative studies is to really give you a window into the black box for process variables.”

Types of study designs mentioned for inclusion in reviews of multicomponent interventions included randomized controlled trials, controlled before and after studies, interrupted time series, repeated measures, and qualitative and observational studies. There was an understanding that RCTs and qualitative studies each have very different uses in systematic reviews. RCTs are helpful to evaluate the efficacy of an intervention whereas qualitative studies are useful to understand the context and implementation factors that are important to systems in deciding whether to implement a complex multicomponent intervention. Ultimately, the choice of scope of study designs to be included largely depends on the topic and intent of the review.

Summary

Frameworks Introduce Structural Information for Synthesis

As discussed previously, organizing complex multicomponent interventions according to a framework can facilitate the description and analysis of the evidence base, and may aid in the contextualization of the results. In a given evidence base, it will probably be possible to organize the interventions and comparators in several frameworks, and in practice the analyst will usually select only one.

Studies, or more accurately, specific features of the interventions used in each study, will be grouped. Adopting a framework incorporates extra-evidential information in the synthesis, in that the framework affects at least the structure of the synthesis strategy, and perhaps more aspects thereof. We outline several approaches to synthesis in the following paragraphs.

Overall Effectiveness of a Class of Interventions: Holistic Approach

Provided that the included studies are similar enough and provide adequate data for meta-analysis, it is straightforward to obtain a summary estimate for the effectiveness of “any intervention” compared with “usual care” using standard meta-analysis methods. The exact statistical approach would depend on the type of outcome and features of included studies. In general, a random effects model should be used to combine studies, and heterogeneity among studies should be routinely assessed.

The “usual care” often does not constitute a homogenous comparison group, and typically there is no common standard across studies so the results of comparisons should be interpreted in the context of this potential heterogeneity—the extent of heterogeneity should still allow a meaningful combined overall estimate. Further, when focusing on an overall class effect, other characteristics of the intervention and the setting are not accounted for. Thus, a summary on the effectiveness of any intervention compared with usual care is probably not easy to interpret and contextualize, and reviewers may favor a more detailed exploration that accounts for the characteristics in the intervention, setting, implementation process or other features. Because frameworks differ in how they organize the observed interventions, the corresponding analytic approaches can also differ.

Effectiveness of Mutually Exclusive Subgroups of Interventions

It may be possible to assign studies to subgroups on the basis of characteristics of the intervention. In the example of pulmonary rehabilitation interventions for chronic obstructive pulmonary disease mentioned previously, rehabilitation programs were organized into three exclusive subgroups, namely programs including skeletal muscle exercise training; programs including only respiratory muscle training; and programs with only educational, behavioral, or psychosocial components.⁴³

A simple approach may be to obtain a summary estimate for each subgroup with separate meta-analyses within each subgroup, or to do a (random intercept) meta-regression analysis with covariates encoding subgroup participation. An advantage of the meta-regression is the flexibility to choose a common heterogeneity parameter for all subgroups (parameter sharing), which can be a pragmatic option if subgroups include small numbers of studies. With subgroup analysis or the meta-regression, the subgroups are treated as fixed factors.

If there are commonalities among the subgroups, a more involved approach could be used by treating subgroups as random. That is, a hierarchical meta-regression could be used. Within each subgroup, the estimated effect sizes follow a random distribution; then at the subgroup level, the mean effects of subgroups also follow a random distribution. In addition to estimating an effect for each subgroup and making comparison among subgroups, an overall effect across the subgroups can be estimated. This approach may be difficult to converge if the number of subgroups is small.⁴⁴ Regardless of whether subgroups are treated as fixed or random factors, analyses will be usually implemented in a regression framework. This means that exploring associations between intervention effects and study characteristics other than those defining the subgroups is relatively straightforward. Expert statistical input should be used in making choices about the modeling, and in specifying meta-regressions, especially for the hierarchical model.

Effectiveness of Interventions by Presence or Absence of Intervention Characteristics (Components) That Can Co-Occur

Reviews may consider interventions to be composites of “components” that can be mixed-and-matched. Because of how components co-occur, it may not be possible to group studies into mutually exclusive and exhaustive subgroups, as in the previous approach. The goal of the synthesis would be to tease out the effect of each component, or of groups of components, based on the observed intervention effects and component combinations in the available studies. In the example mentioned previously for quality improvement strategies to manage diabetes, a systematic review of 162 RCTs characterized the interventions according to presence or absence of 12 predefined components (self-management promotion, team changes, case management, patient education, facilitated relay, electronic patient register, patient reminders, audit and feedback, clinician education, clinician reminders, continuous quality improvement, and financial incentives).⁴⁵ Because the observed combinations of components were almost as many as the included RCTs, it was impossible to draw inferences for each individual intervention. Describing the interventions as mixed-and-matched of components was a pragmatic choice to make the synthesis feasible, but was also supported by stakeholders and experts in this field. [Personal communication with Andrea Tricco and Noah Ivers]

One way to tease out the effect of each component is to model the intervention effect as a function of the components of interest in a meta-regression. The simplest approach would assume that components have only additive effects (they do not interact). However, one could straightforwardly account for interactions between components by adding appropriate interaction terms in the regression. Although meta-regressions allowing all possible interactions between all components could be specified, in practice, models exploring many or high-order interactions will probably be infeasible. This is because the number of parameters in the meta-regression model grows fast as more and higher order interactions are considered, requiring a (very large) number of studies. Further, if, for example, two components always co-occur, it is impossible to identify the main effect of each component or their interaction. More generally, because the observed combinations of components are unlikely to span the space of possible combinations, interaction effects may not always be identifiable.

Exploring the Impact of Implementation Factors and Other Factors

Implementation factors are particularly important for multicomponent interventions. If there are enough studies available, impact of implementation factors such as adherence to the program model, exposure or dosage, or quality of program service delivery could be evaluated using meta-regression if these factors could be adequately quantified.

As mentioned already, it is relatively straightforward to include such factors in meta-regressions for all aforementioned analytic choices. It is probably a good idea to request expert statistical input for the exact specification of the meta-regression models.

Key Informant Interviews

KIs repeatedly spoke about the challenge in analyzing information about complex multicomponent interventions (Table C-4, Appendix C). At one extreme, performing no analysis whatsoever was considered suboptimal:

…a lot of reviewers end up doing when they look at complex interventions is they just sort of wave their hands in the air and they say, ‘We're just going to do a descriptive analysis, a narrative analysis, a qualitative analysis.’ Typically what that means is they have no idea how they're going to do the analysis and they just either do vote counting and they make some inappropriate analysis based on the number of positive and negative studies, which is usually misinformative, or they just list the results of one study after another.

Similarly, performing quantitative analyses without clinical reason for groupings was also thought to hold limited value: “I guess my concern is trying to take a kind of blind approach to grinding the data out and hoping that insights will sort of fallout from a quantitative analysis I think probably it's unlikely to be successful.” KIs often mentioned the importance of including subject matter experts in the review team or consulting with clinicians to understand the clinical reasoning that would allow meaningful groupings of data for analysis: “I think that (adequate structure) really demands very thoughtful reviewers who have strong subject matter expertise, who have the ears and the input of researchers and practitioners to really make sure that they're combining apples and oranges, but not Volkswagens.” Suggestions were made to borrow analytic approaches from fields such as education and econometrics as these fields were thought to have longer experience with the mixed study designs encountered in this field.

Summary

Overall, attempting to conduct the most rigorous analysis possible is important for reviews to inform and change a field. The “correct” synthesis approach is unknowable without knowing the context of the particular subject of the review. Analysts must justify their choices, and may wish to consider alternatives in sensitivity or robustness analyses, especially for choices that rely heavily on extra-evidentiary assumptions. Synthesis of effects cannot transpire, however, unless the analyst can find a level of abstraction where (1) between-study differences can be reasonably ignored and (2) the commonalities are interpretable and nontrivial. The analyst must be aware that choices of framework inform the synthesis and can affect conclusions. Ultimately, the analytic approach for multicomponent interventions depends upon decisions made for approaching a review of a given topic. Considerations about lumping or splitting, mixing qualitative and quantitative studies, and heterogeneity across studies must be considered both in the approach to the topic and in analytic considerations. Having a good understanding of the topic area and needs of users is critical in order to understand key components, potential interactions, and synergy among dimensions, which are all essential to determine the appropriate analytic approach.

Key Informant Interviews

As the field of systematic reviews of complex, multicomponent interventions continues to develop, reviews can be of increasing value to consumers. Reviewers commented that they are often limited by inconsistent reporting of informational details among individual studies. They commented that the field would be greatly advanced if there was a shared understanding among researchers and reviewers for what primary research and reviews should report when presenting results of multicomponent interventions. Their comments reflect this below:

…we usually don't get enough information to know if they are transferable or not, and it's because of the missing implementation details.

The bad thing is that sometimes it's just that they made different decisions rules and they are reporting everything, and you can't compare it because the metrics are all over the place. So, yes, I think that from the primary study perspective really making sure that there is like a common set of what needs to be reported and maybe even used in terms of outcome, in terms of what you say about process.

I don't think you could come up with a comprehensive list, but I think you could come up with some expectations of the minimum sorts of information you could expect to find.

What occurred to us is if in this discussion AHRQ and say the broader community comes up with the dimensions that really need to be addressed in any delivery system intervention so that we ended up with some standardization of the descriptions of the interventions that way at least if it's a negative intervention or a preliminary intervention it's at least clear to the broader community and to other implementers what the heck it was, and they could sort of implement it. And our sense, at least from the various reviews that we’ve done, is that presently there is no such set of standards.

…that was actually one of our recommendations, too, was that there ought to be a requirement that folks have sort of a detailed description that was available to anybody who wanted to consider implementing that intervention.

We asked KIs to list those elements that are critical for primary researchers and reviewers to report. Their responses of standard reporting elements are listed in Table 3. We reviewed existing reporting standards to understand the degree to which reporting elements suggested from KIs had been previously addressed. We compared these to several existing standards for reporting from the EQUATOR network⁴⁶ and one from the REFLECTION network.⁴⁷ Three instruments adapted from CONSORT and one instrument for reporting evaluations of Quality Improvement initiatives are relevant to complex multicomponent interventions, and one proposed instrument specifically addresses reporting standards for complex interventions. The last column in Table 3 provides a cross reference of element reported by CONSORT, SQUIRE, CReDECI, or other sources with elements mentioned by KIs.

Table 3

Key informant recommendations for reporting elements in primary research of complex multicomponent interventions.

The first adaptation of CONSORT reporting guidelines for RCTs to special topics was done for behavioral medicine research reporting in 2003.⁴⁸ This group specified behavioral medicine research applications across all initial CONSORT criteria (Items 1–22),⁴⁹ with expanded consideration of the minimal details for behavioral medicine interventions and controls (Item 4) across nine dimensions: content, elements, targeting or tailoring of the intervention; provider; format; setting; intervention recipient; intervention target; intensity; duration; and intervention incentives. Five novel items were suggested for behavioral medicine RCT reporting including: training of treatment providers (Item 23); supervision of treatment providers (Item 24); patient and provider treatment allegiance or preference (Item 25); treatment delivery integrity (Item 26); and treatment adherence (Item 27). Throughout the adaptation to behavioral medicine, there was an emphasis on communication to allow the reader to judge the credibility of the trial result, to consider how applicable the findings were to his/her setting, and to potentially replicate the findings (in practice or in confirmatory research). In 2004, a complementary effort through public health focused on nonrandomized evaluations of behavioral and public health interventions.⁵⁰ This approach also adapted the 22-item initial CONSORT approach to nonrandomized designs and to behavioral/public health interventions, addressing issues such as recruitment methods, intervention characteristics, and measurement issues, also raised by Davidson, et al. In 2008, the CONSORT statement was formally extended to reporting RCTs of nonpharmacologic treatments.⁵¹ This updated statement expanded the intervention components (Item 4) to include tailoring, standardization, and care provider adherence, with emphasis on considering the characteristics of care providers and participating centers known to influence outcomes. A new item considering the details of implementation as achieved was also added.

In 2008, SQUIRE suggested standard elements to consider when reporting evaluations of planned quality and safety of care initiatives.⁵² While adapting the general approach of CONSORT (i.e., using the traditional sections of a scientific paper to organize the reporting recommendations), the guidance was more qualitative, emphasizing principles for deciding what to report (i.e., “Specify who [champions, supporters] and what [events, observations] triggered the decision to make changes, and why now [timing],” in addition to suggesting specific descriptive items to include).

In 2011, CReDECI⁵³ proposed criteria for reporting the development and evaluation of complex interventions, to be eventually developed for a reporting statement. It uses the MRC framework³ as a main source to include items specific to complex interventions and takes into account the use of different study designs in the development and evaluation. For this reason the proposed elements of CReDECI are organized by the stages outlined in the MRC framework rather than sections of a publication. These stages are development, feasibility and pilot testing, and introduction of the intervention and evaluation. This captures the concept of time; the expected and actual aspects of the intervention, context, and their interaction; and differences in intent of a study, such as piloting or implementation. The elements are not prescriptive or exhaustive in the type of detail to report, and may be applicable to the broader variety of study types used to evaluate complex interventions.

The CONSORT adaptations have largely arisen out of a need to fine tune the reporting requirements for interventions that have critically important implementation details different from pharmacological treatments. The challenge in reporting multicomponent complex interventions may not simply be a question of focus or degree of intervention detail, but rather that of a more applicable paradigm. As others have pointed out, complex multicomponent interventions are more than a defined set of components predictably producing outcomes; instead, their impact is highly dependent on human behavior for delivering or responding to complex interventions, and these human actions are highly influenced by context.²³ In some ways, the quality improvement approach in SQUIRE may be more applicable to developing a standardized reporting framework for multicomponent complex interventions; such an approach would involve specifying what any reader would generally need to know in order to understand, critically assess, and act on the findings. Specific intervention or setting reporting details would be warranted, but more importantly would need to be placed in the context. Articulating the important issues or dimensions of context that may impact the intervention is likely more critical than a detailed listing of very specific characteristics of interventions without context. CReDECI does include the reporting of the purported, actual, and unexpected influences of context during development and in evaluation.