Table C-3, Study Design - Systematic Reviews of Complex Multicomponent Health Care Interventions

“In most of the Effective Practice and Organisation Care work, we've limited reviews to randomized trials, controlled before or after studies, and interrupted time series. We've also included, I suppose, non-randomized trials and repeated measures, which are a variance of those. We have very little experience with other types of study designs and we're just starting, colleagues of mine, to look at ways of incorporating findings from qualitative research. They (qualitative studies) informed judgments about the applicability of the intervention and the feasibility of the intervention primarily. To some extent, they informed other judgments.”
“… it's especially a problem with smaller studies and single site studies, because what they don't account for is the enthusiasm factor. I think that's why so many of them are non-reproducible, because in fact the biggest part of the intervention was often the implementers rather than necessarily even the intervention.”
“We've all been taught pre-posts are terrible, but I think in some situations if they're carefully done they could be okay.”
“Clearly there's some case series where the case fatality rate before and after the intervention is so dramatically different that the odds that the intervention made the difference seem to be very high.”
“I certainly think the net should extend beyond randomized control trials to include well designed observational studies, and the criteria for inclusion should be based on specific characteristics of the study, not on what broad class of design they fall within.”
“Case study designs could give useful information, not so much necessarily about effect, but certainly about issues around implementability around contextual factors which might affect the uptake of particular interventions.”
“If the question is about the effectiveness of the intervention, then experimental designs are better than observational ones… If the question is around, ‘Well, what are the components? Why do they seem to work?’ Then observational qualitative kinds of designs would be the appropriate ones to help try to tease that out.”
“…say we found 400 studies and we looked at the three that were randomized trials of XXX. I just think on its face value that looks ridiculous.”
“There is a classic text though, to my mind at least, that really is talking about study design. It's called Realistic Evaluation. It's by two British statisticians, Pawson and Tilley, and what they do is criticize randomized designs. They criticize them explicitly because they lose context, and then recommend new designs where you include the context as part of your design, build it right in as part of your design… they base their what they call CMO designs, context, mechanism, outcomes designs.”
“… that we need a much better understanding of when it's useful to look for these observational studies because it's very time-consuming to do so. It adds a lot of work to the review, and if you have 35 trials and 1 controlled or after study, it's a very unlikely that the control or after study is going to make an important contribution to the findings of that review even though you've had to wade through 5,000 abstracts to find it. I think where there are trials, I'm keen to stick to them and I think we need better—more information on where it's useful to look for other things.”
“I'm also a bit ambivalent about some of the more complex designs like … difference and all of that stuff—some of those more econometric designs. There's a lot of interest in those now… but I think realistically most review authors are going to struggle to manage those kinds of data. They're very complicated to understand and to analyze in this analysis.”
Role of Qualitative Studies
“There are a whole lot of ways that qualitative evidence … can contribute. I think they could be helpful in developing better definitions on an intervention and understanding implementation considerations and understanding why interventions work or not in helping to see what outcomes are seen as important by different stakeholders.”
“We have very little experience with other types of study designs and we're just starting, colleagues of mine, to look at ways of incorporating findings from qualitative research. They (qualitative studies) informed judgments about the applicability of the intervention and the feasibility of the intervention primarily. To some extent, they informed other judgments. For process evaluations, it's often qualitative research.”
“I think the argument that's being made here is the need for mixed methods studies, so that there's a qualitative component, because we found the same thing, you miss a lot if you just solely rely on existing literature, and even just preconceived notions based on experience.”
“… qualitative evaluation, I often argue, is more important because what you want to understand is what went wrong. If you're producing negative studies on a repeated basis, you really do need those qualitative observations to understand why the study was negative.”
“I think some of the most interesting information that's come out of it has come out of qualitative discussions, focus groups, and feedback from the clinicians in the practices talking about what they liked about the intervention. What their patients liked, and also, talking about the real challenges of being a busy primary care clinician with lots of competing demands for their time and attention, but struggling to try to make something that doesn't fit within the usual—”
“I think qualitative studies can be very helpful in terms of learning about how to implement something or they can give insight into why something works or doesn't work.”

“In most of the Effective Practice and Organisation Care work, we've limited reviews to randomized trials, controlled before or after studies, and interrupted time series. We've also included, I suppose, non-randomized trials and repeated measures, which are a variance of those. We have very little experience with other types of study designs and we're just starting, colleagues of mine, to look at ways of incorporating findings from qualitative research. They (qualitative studies) informed judgments about the applicability of the intervention and the feasibility of the intervention primarily. To some extent, they informed other judgments.”

“… it's especially a problem with smaller studies and single site studies, because what they don't account for is the enthusiasm factor. I think that's why so many of them are non-reproducible, because in fact the biggest part of the intervention was often the implementers rather than necessarily even the intervention.”

“We've all been taught pre-posts are terrible, but I think in some situations if they're carefully done they could be okay.”

“Clearly there's some case series where the case fatality rate before and after the intervention is so dramatically different that the odds that the intervention made the difference seem to be very high.”

“I certainly think the net should extend beyond randomized control trials to include well designed observational studies, and the criteria for inclusion should be based on specific characteristics of the study, not on what broad class of design they fall within.”

“Case study designs could give useful information, not so much necessarily about effect, but certainly about issues around implementability around contextual factors which might affect the uptake of particular interventions.”

“If the question is about the effectiveness of the intervention, then experimental designs are better than observational ones… If the question is around, ‘Well, what are the components? Why do they seem to work?’ Then observational qualitative kinds of designs would be the appropriate ones to help try to tease that out.”

“…say we found 400 studies and we looked at the three that were randomized trials of XXX. I just think on its face value that looks ridiculous.”

“There is a classic text though, to my mind at least, that really is talking about study design. It's called Realistic Evaluation. It's by two British statisticians, Pawson and Tilley, and what they do is criticize randomized designs. They criticize them explicitly because they lose context, and then recommend new designs where you include the context as part of your design, build it right in as part of your design… they base their what they call CMO designs, context, mechanism, outcomes designs.”

“… that we need a much better understanding of when it's useful to look for these observational studies because it's very time-consuming to do so. It adds a lot of work to the review, and if you have 35 trials and 1 controlled or after study, it's a very unlikely that the control or after study is going to make an important contribution to the findings of that review even though you've had to wade through 5,000 abstracts to find it. I think where there are trials, I'm keen to stick to them and I think we need better—more information on where it's useful to look for other things.”

“I'm also a bit ambivalent about some of the more complex designs like … difference and all of that stuff—some of those more econometric designs. There's a lot of interest in those now… but I think realistically most review authors are going to struggle to manage those kinds of data. They're very complicated to understand and to analyze in this analysis.”

Role of Qualitative Studies

“There are a whole lot of ways that qualitative evidence … can contribute. I think they could be helpful in developing better definitions on an intervention and understanding implementation considerations and understanding why interventions work or not in helping to see what outcomes are seen as important by different stakeholders.”

“We have very little experience with other types of study designs and we're just starting, colleagues of mine, to look at ways of incorporating findings from qualitative research. They (qualitative studies) informed judgments about the applicability of the intervention and the feasibility of the intervention primarily. To some extent, they informed other judgments. For process evaluations, it's often qualitative research.”

“I think the argument that's being made here is the need for mixed methods studies, so that there's a qualitative component, because we found the same thing, you miss a lot if you just solely rely on existing literature, and even just preconceived notions based on experience.”

“… qualitative evaluation, I often argue, is more important because what you want to understand is what went wrong. If you're producing negative studies on a repeated basis, you really do need those qualitative observations to understand why the study was negative.”

“I think some of the most interesting information that's come out of it has come out of qualitative discussions, focus groups, and feedback from the clinicians in the practices talking about what they liked about the intervention. What their patients liked, and also, talking about the real challenges of being a busy primary care clinician with lots of competing demands for their time and attention, but struggling to try to make something that doesn't fit within the usual—”

“I think qualitative studies can be very helpful in terms of learning about how to implement something or they can give insight into why something works or doesn't work.”

From: Appendix C, Selected Quotes From Interviews

Contents

Bookshelf

Systematic Reviews of Complex Multicomponent Health Care Interventions [Internet].

Table C-3Study Design

Views

Other titles in these collections

Recent Activity