Background:

Many steps in the preparation of effectiveness and comparative effectiveness reviews (CERs) by the Agency for Healthcare Research and Quality’s Effective Health Care (EHC) Program involve outside review and input (e.g., by Key Informants, the Technical Expert Panel, Draft Report Peer Reviewers). However, development of bibliographic database search strategies is currently not consistently peer reviewed. An opportunity exists for the EHC Program to leverage its expert searchers across the 14 Evidence-based Practice Centers (EPCs) in the Program by implementing a process of peer review of search strategies.

Objective:

Due to the number and frequency of observed errors in published search strategies uncovered in the research literature, an evaluation of the feasibility of instituting a peer review process is warranted.

Specific Aims:

1. Evaluate whether the PRESS instrument or no-instrument (‘free-form’ evaluations) is preferred by Technical Expert Peer Reviewers (TEPRs) of search strategies.
2. Evaluate the usefulness of a peer review process for database search strategies, that is, do peer reviews change search strategies.
3. Evaluate the costs of implementing a formal peer review of search strategies program as a part of the review process.

Methods:

We identified current research protocol phase CER search strategies to review, TEPRs from across the EHC Program, and the original expert searchers who worked on developing each of the search strategies. Each TEPR was assigned to either the control group who only wrote “free-form” reviews or to the group who initially wrote a “free-form” review, then trained using the PRESS Instrument, and finally completed the last review using the PRESS instrument. Original expert searchers were asked to comment on the reviews of their searches. One study researcher administered the peer review process and log time required to perform it, while the other, blinded study researcher analyzed qualitative and quantitative data derived from the reviews, as well as demographic information about the TEPR and original expert searchers.

Results:

As a whole, the group of people available to conduct and review search strategies in the EHC Program is professionally educated and very experienced. Of the 24 respondents to the demographic survey, 20 (83%) have master’s degrees in library science. Most have more than 10 years’ experience as a librarian or other information professional. Sixty-seven percent of respondents have more than 5 years experience contributing to systematic reviews, with 63 percent having contributed to more than 10 systematic reviews. These experienced searchers have a variety of relationships with the EPCs for whom they do searches. Of the 25 peer reviewers invited to participate, 24 completed the initial free-form round of reviews, and 15 completed either the PRESS review or a second free-form review as part of our control group. For the most part peer reviewers were positive about the review process, although many hesitated to incorporate the review process into their current workflow. All of the reviewers found the background material (systematic review protocol) helpful to the review. Of those who used the PRESS instrument, 82 percent (9) indicated that the instrument was helpful, 18 percent (2) reported that it was neither helpful nor limiting, and none of the reviewers indicated that the PRESS instrument was limiting. The PRESS instrument reviews contained more recommendations on the whole and in particular had more comments that could be termed error detection—specific comments about spelling or syntax indicating that a mistake had been made. In 97 percent of cases, the original searcher indicated that the comments did not cause them to alter their search strategies.

Conclusions:

While the results of this study suggest that if a formal peer review process is to be valuable then it would need to be both timely and timed for a window of opportunity immediately prior to the finalization of the protocol. Even if a formal peer review process is not implemented, the PRESS instrument could be useful in informal peer review or even self review. If review of search strategies is to take place, then these results suggest that the use of the PRESS instrument would cut down the time taken, increase the likelihood of response and be more effective in identifying actual errors in search strategies. Additionally, the content of the reviews indicates that there are several search tactics for which there is no consensus, and further research could help us to understand variation in practice around such issues as limits, searching for observational studies, and searching for outcomes and comparators. The process of reviewing other searchers’ work can bring these issues to light, and a peer review-like process could be used to start investigations and discussions of what techniques work and why. Finally, many of the reviewers commented on the difficulty of reading the search strategies as currently presented. The EHC Program currently has no standards for reporting search strategies, and there is no recognized standard for reporting search strategies. Adopting standard of reporting designed to facilitate review may make it easier to review search strategies both internally and when reported to the public.