Our tool was initially based on earlier instruments, expert panel input, cognitive testing, and face validity testing. This project builds on our earlier work by revising the questions, providing further instructions to users, and proposing a framework for considering confounding across a body of observational study evidence.

Specifically, we made further modifications and enhancements to the RTI Item Bank, including: eliminating an item asking reviewers to designate a study as prospective or retrospective because it was considered uninformative, eliminating items evaluating precision because it is generally no longer coupled with risk of bias assessment, modifying item bank questions so that they all focus on identifying study implementation characteristics that would increase the risk of bias, and enhancing the instructions provided to reviewers who use the tool.

In addition, we propose a substantial expansion in the critical appraisal of confounding when systematic reviews include observational studies for evaluation of benefits of interventions or their harms. Attributing causality to interventions from such evidence requires prespecification of anticipated sources of confounding prior to the review, followed by appraisal of potential confounders at three levels: outcomes, studies, and the body of evidence. During the review, additional insights may emerge. These tasks require a substantial commitment of senior researchers on each systematic review team who are deeply knowledgeable about the topic to conduct tasks that may have sometimes been previously assigned to junior team members. This expanded focus on confounding serves two purposes: first, it helps to explain results when heterogeneous bodies of evidence turn up conflicting results; and second, it helps to undergird the validity of the results of the systematic review, regardless of the level of heterogeneity. The latter may be particularly important in instances when a body of evidence may be relatively homogeneous in showing an effect but individual studies all fail to account for common confounders.

Establishing consensus requires collaboration across a wide range of expertise. This methods project benefited from a diverse set of Working Group members with varying experience, expertise, and perspectives. The Working Group members were particularly engaged in discussions on how to think about the issue of confounding over the course of a systematic review. Our project suffered, however, from the format we employed to engage Working Group members in revising questions for the RTI Item Bank: the length, frequency, and format of communication (telephone and email) did not permit a sustained exchange of ideas. We believe that the intensity of this effort requires one or more in-person meetings to establish consensus on needed items for specific research designs.