In the first phase of this project, we aimed to determine the reliability in study quality assessments across different reviewers. We also examined different aspects of the study quality assessment process, focusing only on randomized controlled trials.

This initial pilot study was guided by the following questions:

1. What is the extent of concordance in quality rating assignments by independent reviewers?
2. What is the variability in the assessment of specific methodological components of studies? Are there components that vary more in their interpretation than others?
3. Are there specific methodological components that affect reviewers’ assessments of study quality ratings but are not captured by commonly used quality item checklists? What are some of the implicit considerations (other than those related to the checklist items) that helped to determine a study’s quality rating?
4. If methods to reduce existing variations in quality assessments are implemented, what is the extent of residual variation in quality assessments following such interventions?
5. What are the relative contributions of individual quality components to the overall assigned rating of a study? Which methodological components are the most influential in determining the final rating of an individual study?
6. Are assessments of individual quality items influenced by the citation information or the publication format of a study under evaluation? Are overall quality ratings of studies sensitive to changes of individual items?

Addressing these questions would provide an appreciation of the subjective elements in reviewers’ thought processes when assessing a study’s quality and also could provide information on the relative contributions of such elements to overall quality rating. Results from this study will be used to plan the next phase.