NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.
Headline
This document provides evidence on how measurement procedures in trials of interventions to improve health produce risk of bias, and provides recommendations on how best to minimise such bias.
Abstract
Background:
Measurement can affect the people being measured; for example, asking people to complete a questionnaire can result in changes in behaviour (the ‘question–behaviour effect’). The usual methods of conduct and analysis of randomised controlled trials implicitly assume that the taking of measurements has no effect on research participants. Changes in measured behaviour and other outcomes due to measurement reactivity may therefore introduce bias in otherwise well-conducted randomised controlled trials, yielding incorrect estimates of intervention effects, including underestimates.
Objectives:
The main objectives were (1) to promote awareness of how and where taking measurements can lead to bias and (2) to provide recommendations on how best to avoid or minimise bias due to measurement reactivity in randomised controlled trials of interventions to improve health.
Methods:
We conducted (1) a series of systematic and rapid reviews, (2) a Delphi study and (3) an expert workshop. A protocol paper was published [Miles LM, Elbourne D, Farmer A, Gulliford M, Locock L, McCambridge J, et al. Bias due to MEasurement Reactions In Trials to improve health (MERIT): protocol for research to develop MRC guidance. Trials 2018;19:653]. An updated systematic review examined whether or not measuring participants had an effect on participants’ health-related behaviours relative to no-measurement controls. Three new rapid systematic reviews were conducted to identify (1) existing guidance on measurement reactivity, (2) existing systematic reviews of studies that have quantified the effects of measurement on outcomes relating to behaviour and affective outcomes and (3) experimental studies that have investigated the effects of exposure to objective measurements of behaviour on health-related behaviour. The views of 40 experts defined the scope of the recommendations in two waves of data collection during the Delphi procedure. A workshop aimed to produce a set of recommendations that were formed in discussion in groups.
Results:
Systematic reviews – we identified a total of 43 studies that compared interview or questionnaire measurement with no measurement and these had an overall small effect (standardised mean difference 0.06, 95% confidence interval 0.02 to 0.09; n = 104,096, I2 = 54%). The three rapid systematic reviews identified no existing guidance on measurement reactivity, but we did identify five systematic reviews that quantified the effects of measurement on outcomes (all focused on the question–behaviour effect, with all standardised mean differences in the range of 0.09—0.28) and 16 studies that examined reactive effects of objective measurement of behaviour, with most evidence of reactivity of small effect and short duration. Delphi procedure – substantial agreement was reached on the scope of the present recommendations. Workshop – 14 recommendations and three main aims were produced. The aims were to identify whether or not bias is likely to be a problem for a trial, to decide whether or not to collect further quantitative or qualitative data to inform decisions about if bias is likely to be a problem, and to identify how to design trials to minimise the likelihood of this bias.
Limitation:
The main limitation was the shortage of high-quality evidence regarding the extent of measurement reactivity, with some notable exceptions, and the circumstances that are likely to bring it about.
Conclusion:
We hope that these recommendations will be used to develop new trials that are less likely to be at risk of bias.
Future work:
The greatest need is to increase the number of high-quality primary studies regarding the extent of measurement reactivity.
Study registration:
The first systematic review in this study is registered as PROSPERO CRD42018102511.
Funding:
Funded by the Medical Research Council UK and the National Institute for Health Research as part of the Medical Research Council–National Institute for Health Research Methodology Research Programme.
Contents
- Plain English summary
- Scientific summary
- Chapter 1. Introduction
- Chapter 2. Measurement reactivity and risk of bias
- Chapter 3. Research informing the development of recommendations
- Asking questions changes health-related behaviour: an updated systematic review and meta-analysis of randomised controlled trials
- Further evidence reviews to inform development of the recommendations
- Delphi procedure to inform the scope of the recommendations
- Expert workshop to develop content of the recommendations
- Chapter 4. Recommendations
- Identify whether or not measurement reactivity is likely to be a major source of bias for a new trial
- Collect further data to inform decisions about whether or not there is risk of bias resulting from measurement reactivity
- Potential actions to minimise risk of bias from measurement reactivity within a trial
- Chapter 5. Future research
- Recommendation 1: more primary research to quantify extent of measurement reactivity
- Recommendation 2: research priorities for studies within a trial
- Recommendation 3: more systematic reviewing to quantify extent and variability of measurement reactivity
- Recommendation 4: the need to better theorise when and why measurement reactivity is likely to occur
- Acknowledgements
- References
- Appendix 1. The MERIT Collaborative Group
- Appendix 2. Example of an interaction between baseline measurement and an intervention in a Solomon four-group design
- Appendix 3. Search strategy for rapid review of systematic reviews
- Appendix 4. Overlap between four recent reviews of the question–behaviour effect
- Appendix 5. Explanatory guide to support Table 4
- Glossary
- List of abbreviations
About the Series
Declared competing interests of authors: David P French was a member of the National Institute for Health Research (NIHR) Public Health Research Funding Board (2015–19). Andrew Farmer is Director of the NIHR Health Technology Assessment programme (2020 to present) and is an NIHR Senior Investigator. Martin Gulliford was a member of the NIHR Health Services and Delivery Research (HSDR) Funding Committee (2016–19). Louise Locock was a member of the NIHR HSDR Funding Committee (2014–19).
Article history
This issue of the Health Technology Assessment journal series contains a project commissioned by the MRC–NIHR Methodology Research Programme (MRP). MRP aims to improve efficiency, quality and impact across the entire spectrum of biomedical and health-related research. In addition to the MRC and NIHR funding partners, MRP takes into account the needs of other stakeholders including the devolved administrations, industry R&D, and regulatory/advisory agencies and other public bodies. MRP supports investigator-led methodology research from across the UK that maximises benefits for researchers, patients and the general population – improving the methods available to ensure health research, decisions and policy are built on the best possible evidence.
To improve availability and uptake of methodological innovation, MRC and NIHR jointly supported a series of workshops to develop guidance in specified areas of methodological controversy or uncertainty (Methodology State-of-the-Art Workshop Programme). Workshops were commissioned by open calls for applications led by UK-based researchers. Workshop outputs are incorporated into this report, and MRC and NIHR endorse the methodological recommendations as state-of-the-art guidance at time of publication.
The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The HTA editors and publisher have tried to ensure the accuracy of the authors’ report and would like to thank the reviewers for their constructive comments on the draft document. However, they do not accept liability for damages or losses arising from material published in this report.
Last reviewed: October 2019; Accepted: June 2020.
- †
The MERIT Collaborative Group are listed in Appendix 1.
- NLM CatalogRelated NLM Catalog Entries
- Reducing bias in trials from reactions to measurement: the MERIT study including...Reducing bias in trials from reactions to measurement: the MERIT study including developmental work and expert workshop
- Evaluation of venous thromboembolism risk assessment models for hospital inpatie...Evaluation of venous thromboembolism risk assessment models for hospital inpatients: the VTEAM evidence synthesis
- Use of drug therapy in the management of symptomatic ureteric stones in hospital...Use of drug therapy in the management of symptomatic ureteric stones in hospitalised adults: a multicentre, placebo-controlled, randomised controlled trial and cost-effectiveness analysis of a calcium channel blocker (nifedipine) and an alpha-blocker (tamsulosin) (the SUSPEND trial)
- Interventions based on early intensive applied behaviour analysis for autistic c...Interventions based on early intensive applied behaviour analysis for autistic children: a systematic review and cost-effectiveness analysis
- Defining the optimum strategy for identifying adults and children with coeliac d...Defining the optimum strategy for identifying adults and children with coeliac disease: systematic review and economic modelling
Your browsing activity is empty.
Activity recording is turned off.
See more...