Recruitment of Technical Experts and Peer Reviewers

At the beginning of the project, we recruited a panel of internal and external technical experts to give input on key steps including the selection and refinement of the questions to be examined. The panel included two internal technical experts from the Johns Hopkins University who have strong expertise in various aspects of the efficacy and/or safety of multivitamins/minerals and evidence-based medicine, and external experts who have strong expertise in nutritional research (see Appendix A ^a). In addition to this panel of technical experts, we recruited a few additional experts to serve as peer reviewers of the evidence report, as described further in the section on Peer Review.

Key Questions

We worked with the technical experts and representatives of OMAR and the Agency for Healthcare Research and Quality (AHRQ) to develop the Key Questions that are presented in the Specific Aims section of Chapter 1 (Introduction). We expanded the preliminary questions to include functionally related nutrient pairs, tuberculosis, hepatitis C, and pulmonary disease, and limited the questions involving efficacy to randomized controlled trials. The Key Questions focus on the efficacy of multivitamins/minerals (and specific single nutrients and functionally related pairs) in the prevention of chronic diseases and conditions as well as the safety of multivitamin/minerals and specific nutrients.

Literature Search Methods

Searching the literature included the steps of identifying reference sources, formulating a search strategy for each source, and executing and documenting each search. Additionally, we searched for medical subject heading (MeSH) terms that were relevant to the specific nutrients and diseases specified in Key Question 1 to help develop the search strategy. We used a systematic approach for searching the literature to minimize the risk of bias in selecting articles for inclusion in the review. In this systematic approach, we had to be very specific about defining the eligibility criteria for inclusion in the review. The systematic approach was intended to help identify gaps in the published literature. We used a systematic approach for extracting data from the studies to minimize the risk of bias in how we extracted data from eligible studies. By creating standardized forms for data extraction, we sought to maximize consistency in identifying all pertinent data available for synthesis.

Organization and tracking of literature search

The results of the searches were downloaded and imported into ProCite^® version 5 (ISI ResearchSoft, Carlsbad, CA). From ProCite, the articles were uploaded to SRS 3.0. We used the duplication check feature in SRS 3.0. This feature allowed us to scan for exact article duplicates, author/title duplicates, and title duplicates. Additionally, this database was used to store citations in portable document format (PDF) and to track the search results at title review, abstract review, article inclusion/exclusion, and data abstraction levels (Figure 2). A list of excluded articles is presented in Appendix D ^a

Figure

Figure 2. Summary of literature search and review process (number of articles ).

Title Review

After the electronic databases were searched, citations were downloaded into ProCite, and uploaded to the SRS 3.0 tracking system. The study team scanned all titles. Two independent reviewers conducted title scans in a parallel fashion. For a title to be eliminated at this level, both reviewers had to indicate that it was ineligible. If the two reviewers did not agree on the eligibility of an article, it was automatically promoted to the next level (see Appendix E ^a, Title Review Form). The title review phase was designed to capture as many studies as possible reporting on the efficacy of single nutrients, related nutrient pairs, and multivitamins/minerals in the primary prevention of chronic diseases and conditions as well as the safety of multivitamins/minerals and a specified set of nutrients. All titles that were thought to address the above efficacy and or safety issues were promoted to the abstract review phase.

Abstract Review

Article Inclusion/Exclusion

Because of the broad array of potentially eligible articles obtained at the abstract review phase, full articles initially selected for review underwent another independent parallel review by investigators to determine if they should be included for full data abstraction. At this phase of review, investigators determined which of the Key Questions each article addressed, and what type of protocol was used in the study (see Appendix E, Article Inclusion/Exclusion Form). If articles were still deemed to have applicable information, they were included in the final article review. Differences in opinions regarding article inclusion or exclusion were resolved through consensus adjudication.

Article Review/Data Abstraction

The purpose of the article review was to confirm the relevance of each article to the research questions, to determine methodological characteristics pertaining to study quality, and to collect evidence that addressed the research questions. Articles eligible for full review could address one or more of the Key Questions. If reviewers determined that an article addressed both efficacy and safety, multiple data abstraction forms were used. We used a systematic approach for extracting data from the studies to minimize the risk of bias in how we extracted data from eligible studies. By creating standardized forms for data extraction, we sought to maximize consistency in identifying all pertinent data available for synthesis.

Each article underwent double review by study investigators for full data abstraction and assessment of study quality. For all data abstracted from studies, we used a sequential review process. In this process, the primary reviewers completed all data abstraction forms. The second reviewer confirmed the first reviewer's data abstraction forms for completeness and accuracy. Reviewer pairs were formed to include personnel with both clinical and methodological expertise. A third reviewer re-reviewed a random sample of articles marked as “ineligible” by the first two reviewers to ensure consistency in the classification of the articles. Reviewers were not masked to the articles' authors, institution, or journal. In most instances, data were directly abstracted from the article. If possible, relevant data were also abstracted from figures. Differences in opinion were resolved through consensus adjudication. For assessments of study quality, each reviewer independently judged study quality and rated items on quality assessment forms. (see Appendix E, Data Abstraction Review Forms)

For all articles containing original data, reviewers extracted information on general study characteristics such as study design, study period and follow up, study participants, sample size, and prior supplement use (see Appendix E, Data Abstraction Review Forms). Data abstracted to the “Arm” forms (see Appendix E, Data Abstraction Review Forms) included: placebo or intervention; nutrients studied; chemical form; dose; units; frequency of use; timing of use; and duration of use.

For studies addressing efficacy (Key Question 1 and/or 3), an outcomes form for efficacy (see Appendix E, Data Abstraction Review Forms) was filled out to obtain the information on study outcomes and adverse effects, and the results from subgroup analyses. Additionally, a specific study quality form was filled out (quality forms were filled out independently) to assess: representativeness of the study population; bias and confounding; description of study supplements/supplementation; adherence and completeness of follow up; statistical analysis; and conflict of interest (see Appendix E, Data Abstraction Review Forms).

Reviewers used an outcomes form to abstract data from articles addressing safety (Key Questions 2 and/or 4) on adverse effects/events and criteria for causality (see Appendix E, Data Abstraction Review Forms).

We also abstracted data from systematic reviews that specifically applied to our Key Questions. This included systematic reviews of calcium and/or vitamin D only, and reviews of studies other than calcium and/or vitamin D only (see Appendix E, Data Abstraction Review Forms).

All information from the article review process was entered into the TrialStat database by the individual completing the review. Reviewers entered comments into the system whenever applicable. The TrialStat database was used to maintain and clean the data, as well as to create detailed evidence tables and summary tables (see Appendix F and Summary Tables).

Quality Assessment

Article quality was assessed differently for different types of studies: efficacy studies (randomized controlled trials only); safety studies; and systematic reviews. The dual, independent review of article quality judged articles on several aspects of each study type's external and internal validity. Quality assessment of studies addressing efficacy included: (1) the representativeness of the study population (description of the study population and where it was drawn, and how well the participants' characteristics were described); (2) bias and confounding (whether this was controlled for in the study design and reported on in the study); (3) description of supplements/supplementation; (4) description of adherence to study protocols and follow up (flow of patients through the study over time, loss to follow up, and participant withdrawal); (5) statistical analysis; and (6) conflict of interest.

Quality assessment of studies addressing safety considered: (1) temporal relationships between timing of supplement use and adverse events (how this was reported); (2) dose-response relationship; (3) whether adverse effects disappeared after supplement use ceased; (4) serum levels of supplements; (5) whether an alternative cause for the adverse event was investigated: and (6) whether the adverse event re-occurred if the supplement was used again.

The quality of each systematic review was assessed using a different set of criteria: (1) whether the question being addressed by the review was clearly stated; (2) comprehensiveness of search methods used and described in the report; (3) whether inclusion/exclusion criteria were clearly defined and appropriate; (4) whether analyses were conducted to measure variability in efficacy; (5) whether study quality was assessed and done appropriately (using validated instruments); (6) whether differences in how outcomes were reported and analyzed across studies were taken into consideration; (7) whether the study methodology was reproducible; and (8) whether conclusions were supported by the data presented.

For each study, we assigned a rating of high, medium or low quality for each domain of study quality based on whether the score for that domain was designated High (80–100%), Medium (50–79%), or Low (0–49%) quality.

Data Synthesis

For each Key Question, we created a set of detailed evidence tables containing all information extracted from eligible studies. The investigators reviewed the tables and eliminated items that were rarely reported. Investigators used the resulting versions of the evidence tables to prepare the text of the report and selected summary tables.

Data Entry and Quality Control

Initial data were abstracted by investigators and entered directly into Web-based data collection forms using; SRS^® 3.0 (TrialStat! Corporation, Ottawa, Ontario, Canada). After a second reviewer reviewed data, adjudicated data were re-entered into Web-based data collection forms by the second reviewer. Second reviewers were generally more experienced members of the research team, and one of their main priorities was to check the quality and consistency of the first reviewers' answers. In addition to the second reviewers checking the consistency and accuracy of the first reviewers, a senior investigator examined all reviews to identify problems with the data abstraction. If problems were recognized in a reviewer's data abstraction, the problems were discussed at a meeting with the reviewers. In addition, research assistants used a system of random data checks to assure data abstraction accuracy.

Grading of the Evidence

At the completion of our review, we graded the quantity, quality and consistency of the best available evidence addressing Key Questions 1 and 3 by adapting an evidence grading scheme recommended by the GRADE Working Group.⁶³ We applied evidence grades to bodies of evidence on each type of nutrient for each major type of outcome. We considered the strength of the study designs with randomized controlled trials considered best, followed by non-randomized controlled trials, observational studies, and case reports. We considered at least two randomized controlled trials reporting on a specific outcome to constitute a body of evidence pertaining to that outcome. If an outcome was evaluated by at least two randomized controlled trials as well as observational studies and case reports, our evidence grade was based only on the randomized controlled trials evaluating that outcome. If an outcome was evaluated by one or no randomized controlled trials, our evidence grade was based on the single randomized controlled trial in addition to the best available non-randomized controlled trial or the best available observational studies (cohort studies considered best, followed by cross-sectional studies and studies with pre-post observational design). We reported the number of studies within the category of best available evidence to assess the quantity of evidence. We also assessed the quality and consistency of the best available evidence, including assessment of limitations to individual study quality (using individual quality scores), certainty regarding the directness of the observed effects in studies, precision and strength of findings, and availability (or lack thereof) of data to answer the Key Question. We classified evidence bodies pertaining to each Key Question into four basic categories: (1) “high” grade (indicating confidence that further research is very unlikely to change our confidence in the estimated effect in the abstracted literature); (2) “moderate” grade (indicating that further research is likely to have an important impact on our confidence in the estimates of effects and may change the estimates in the abstracted literature); (3) “low” grade (indicating further research is very likely to have an important impact on confidence in the estimates of effects and is likely to change the estimates in the abstracted literature); and 4) “very low” grade (indicating any estimate of effect is very uncertain).

Peer Review

Throughout the project, feedback was sought from the technical experts through ad hoc and formal requests for guidance. A draft of the completed report was sent to the technical experts and peer reviewers, as well as to the representatives of the NIH and AHRQ. In response to the comments of the technical experts and peer reviewers, revisions were made to the evidence report, and a summary of the comments and their disposition has been submitted to AHRQ.

Footnotes

a

Appendixes cited in this report are provided electronically at: http://www​.ahrq.gov/clinic​/tp/multivittp.htm