Table 15Difference between responses of interviewees with and without modeling experience

Interview ThemeInterviewees With Experience (N=15)Interviewees Without Experience (N=4)
Attitudes Toward Models and Appropriateness of Modeling in Systematic Reviews
  • Important set of techniques and strategies for analysis and should be incorporated into systematic reviews.
  • Natural extension of the systematic review by addressing gaps in the literature and extending information about intermediate benefits and harms to terminal outcomes.
  • Systematic review should be limited to synthesis and meta-analysis of all available empirical and observational evidence.
  • Models are outside the scope and purpose of the systematic review.
Research Questions and Contexts Best Suited for Decision and Simulation Modeling
  • Comparison of testing strategies (start, stop and interval).
  • Determination of complicated net benefit calculations by linking intermediate to terminal benefits and harms with additional data sources.
  • Questions with high degree of uncertainty.
  • Application of findings to subpopulations not included in original study.
  • Situations with high degree of uncertainty.
  • Difficulty enumerating, but agreed with the “with experience” examples when prompted.
Definition of Decision and Simulation Models
  • Mathematical representation of a decision based on empirical input parameters, supported by a framework, and subject to a set of identifiable assumptions.
  • Confusion on where modeling is defined differently from statistical inference.
Evaluation of Models and Assessment of Model Outcomes
  • Quality and expertise of the modeler(s).
  • Lack of defined standards.
  • Inspection of assumptions and theoretical framework (natural history of disease representation).
  • Focus on the quality and “believability” of the output parameters, and whether multiple models generated similar results.
  • Lacked familiarity with any empirical measures of model quality.
Decision and Simulation Models Results as Evidence
  • Outputs generated from models merit inclusion in systematic reviews as evidence.
  • Modeling offers access to parameters that might not otherwise be available (e.g. subpopulations).
  • Model evidence is “manufactured” or “model produced” and thus must be kept separate from empirical evidence (RCT or observational)
  • There is no evidence grading for model-based parameters.
Impact of Decision and Simulation Modeling on Systematic Reviews
  • Models require additional time and expense, and are not always able to be anticipated at the initiation of a project.
  • Likely to add 20–40% to the time and expense of a typical systematic review.
  • Need a mechanism to include a model after the question refinement phase has been completed.
  • Would require expertise that some EPCs do not have in-house, and thus must contract for externally.
  • Need to have guidelines from the Methods Manual.
Training Needs
  • Increase training opportunities for doctoral and post-doctoral positions to train modelers.
  • Need for seminars and programs to train existing EPC staff.
  • Identify modeling groups with specific expertise to contract with for model components of systematic reviews.

RCT = randomized controlled trial; EPC = Evidence-based Practice Center

From: Use of Modeling in Systematic Reviews: The EPC Perspective

Cover of Decision and Simulation Modeling in Systematic Reviews
Decision and Simulation Modeling in Systematic Reviews [Internet].
Kuntz K, Sainfort F, Butler M, et al.
Copyright Notice

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.