U.S. flag

An official website of the United States government

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Williams JW, Plassman BL, Burke J, et al. Preventing Alzheimer's Disease and Cognitive Decline. Rockville (MD): Agency for Healthcare Research and Quality (US); 2010 Apr. (Evidence Reports/Technology Assessments, No. 193.)

  • This publication is provided for historical reference only and the information may be out of date.

This publication is provided for historical reference only and the information may be out of date.

Cover of Preventing Alzheimer's Disease and Cognitive Decline

Preventing Alzheimer's Disease and Cognitive Decline.

Show details

Appendix DQuality Assessment – RCTs

Quality Assessment – RCTs*

General instructions: Grade each criterion as “Yes,” “No,” “Partially,” or “Can’t tell.” Factors to consider when making an assessment are listed under each criterion. Where appropriate (particularly when assigning a “No,” “Partially,” or “Can’t tell” score), please provide a brief rationale for your decision (in parentheses) in the evidence table.

  1. Were the groups similar at baseline in terms of baseline characteristics and prognostic factors? (Consider baseline prognostic characteristics of intervention/control groups including age, sex, race, educational level, general medical conditions, and performance on a cognitive measure.)
    1. No important baseline differences
    2. Important baseline differences
    3. Can’t tell if important baseline differences (not reported or key baseline characteristics not reported)
  2. Were AD/cognitive outcomes assessed using a valid methodology and criteria? (See details below.)
    1. Valid method used (assessment method and definition)
    2. Valid method used only in some of the subjects
    3. Valid method not used
  3. Were subjects and providers blind to the intervention/exposure status of participants?
    1. Subjects blind to exposure/intervention
    2. Providers blind to exposure/intervention
  4. Were outcome assessors blind to exposure/intervention status?
  5. Were incomplete outcome data adequately addressed? (See more detailed guidance below.)
  6. Was the differential loss to follow-up between the compared groups low (defined as < 10%)?*
    1. *Note: If event rates (e.g., conversion to AD) are low, then even smaller differences in f/u by group could lead to large biases in estimate of effect.
  7. Was the overall loss to follow-up low (defined as < 30%)?
    1. Where different numbers of patients are followed up for different outcomes, use the number followed up for the primary outcome for this calculation.
  8. Conflict of interest reported and insignificant?
    1. Is the source of funding identified?
    2. Is the funding from a source that does not have a vested interest in the study results?
  9. Were the methods used for randomization adequate?
    1. Yes, true random number generator (e.g., computer randomization)
    2. No, not true random number generator (e.g., every other, odd or even DOB, patient record number)
  10. Was allocation concealment adequate? (Allocation sequence should be described in sufficient detail to determine whether intervention allocations could have been foreseen in advance of, or during, enrollment.)
    1. Allocation concealment was adequate (e.g., call central number for intervention allocation after eligibility confirmed, sequentially numbered sealed opaque envelopes, sequentially numbered drug containers of identical appearance)
    2. Allocation concealment inadequate

Detailed guidance for Item 2 – assessment of AD/cognitive outcomes

Principles for an acceptable criterion standard for diagnosis of AD

  1. Uses established diagnostic criteria (e.g., DSM-IV, NINCDS-ADRDA, ICD or similar).
  2. Uses an acceptable method for obtaining the necessary data to apply the diagnostic criteria, defined as an in-person assessment using an assessment battery that addresses the key domains in the diagnostic criteria, namely, at least memory plus one or more of the following: orientation, agnosia, aphasia, apraxia, executive function, or effect on functional status. If the diagnosis is from medical records only, need evidence that a formal in-person evaluation was done to determine the diagnosis of AD.
  3. Pathological specimens alone are not satisfactory since AD is a clinical diagnosis. If AD is diagnosed using an acceptable criterion standard, then pathological specimens could provide useful supplementary information.

Principles for an acceptable criterion standard for diagnosis of cognitive decline

  1. An agreed upon set of diagnostic criteria for a categorical diagnosis (e.g., MCI, CIND, or similar as described in text). The diagnostic criteria will vary across studies, but each study should provide details of the criteria used for MCI, CIND, or similar diagnostic terms for mild cognitive symptoms. The definition should include mild cognitive impairment reported by the individual or informant that did not meet criteria for dementia, or performance on neuropsychological measures that was both below expectation and considered to be in the impaired range based on normative standards.
  2. An acceptable method for obtaining the necessary data to apply the diagnostic criteria, defined as an in-person assessment using an assessment battery that addresses the key domains in the diagnostic criteria, namely, at least memory plus one or more of the following: orientation, agnosia, aphasia, apraxia, executive function, and evidence that cognitive impairment does not significantly interfere with functional status. If the diagnosis is from medical records only, need evidence that a formal in-person evaluation was done to determine the diagnosis of mild impairment.
  3. If cognition is assessed at two or more time points, then change on a validated instrument.

Detailed guidance for Item 5 (“Were incomplete outcome data adequately addressed?”) – taken from Cochrane Handbook for Systematic Reviews of Interventions2, Table 8.5.c

Criteria for a judgment of “Yes” (i.e., low risk of bias)Any one of the following:
  • - No missing outcome data;
  • - Reasons for missing outcome data unlikely to be related to true outcome (for survival data, censoring unlikely to be introducing bias);
  • - Missing outcome data balanced in numbers across intervention groups, with similar reasons for missing data across groups;
  • - For dichotomous outcome data, the proportion of missing outcomes compared with observed event risk not enough to have a clinically relevant impact on the intervention effect estimate; *(see example below)
  • - For continuous outcome data, plausible effect size (difference in means or standardized difference in means) among missing outcomes not enough to have a clinically relevant impact on observed effect size;
  • - Missing data have been imputed using appropriate methods.
Criteria for the judgment of “No” (i.e., high risk of bias)Any one of the following:
  • - Reason for missing outcome data likely to be related to true outcome, with either imbalance in numbers or reasons for missing data across intervention groups;
  • - For dichotomous outcome data, the proportion of missing outcomes compared with observed event risk enough to induce clinically relevant bias in intervention effect estimate; ; *(see example below)
  • - For continuous outcome data, plausible effect size (difference in means or standardized difference in means) among missing outcomes enough to induce clinically relevant bias in observed effect size;
  • - “As-treated” analysis done with substantial departure of the intervention received from that assigned at randomization;
  • - Potentially inappropriate application of simple imputation.
Criteria for the judgment of “Can’t tell” (uncertain risk of bias)Any one of the following:
  • - Insufficient reporting of attrition/exclusions to permit judgment of “Yes” or “No” (e.g., number randomized not stated, no reasons for missing data provided);
  • - Study did not address/report this outcome.

*Example for risk of bias due to incomplete follow-up

Historically, methodologists have sometimes suggested somewhat arbitrary thresholds for acceptable loss to follow-up (e.g. less than 20%). The significance of particular rates of loss to follow-up, however, varies widely and is dependent on the relation between loss to follow-up and number of events. For instance, loss to follow-up of 5% in both intervention and control groups provides little threat to bias if event rates were 20% and 40% in intervention and control groups respectively. If event rates were 2% and 4%, however, concern with 5% loss to follow-up is much greater.

Example where lost to f/u is a relatively low proportion of those with events and little risk of bias. RR=0.5 (.21/.42) and if assumed all lost to f/u had events, RR=0.55 (0.25/0.45)

Enrolled/FU outcomesLost to F/UEvent rateEvent rate if lost to f/u had events
Intervention100/95520/95=.2125/100=.25
Control100/95540/95=.4245/100=.45

Example where lost to f/u is a relatively higher proportion of those with events and significant risk of bias. It only takes a few lost to follow to have had events to change the difference in event rates substantially. RR=0.5 (.02/.04) and if assumed all lost to f/u had events, RR=0.78 (0.07/0.09) and may be distorted further if event rates in the lost to f/u differed between intervention and control

Enrolled/FU outcomesLost to F/UEvent rateEvent rate if lost to f/u had events
Intervention100/9552/95=.027/100=.07
Control100/9554/95=.049/100=.09

References

1.
Agency for Healthcare Research and Quality. Methods Reference Guide for Effectiveness and Comparative Effectiveness Reviews, Version 1.0 [Draft posted Oct. 2007]. Rockville, MD: Agency for Healthcare Research and Quality; [Accessed September 3, 2009]. Available at: http:​//effectivehealthcare​.ahrq.gov/repFiles​/2007_10DraftMethodsGuide.pdf.
2.
Higgins J, Altman D. Higgins J, Green S, editors. Assessing the risk of bias. Cochrane Handbook for Systematic Reviews of Interventions (version 5.0.1) updated September 2008: The Cochrane Collaboration.

Footnotes

*

Taken from AHRQ et al., 20071 and Higgins et al., 2008.2

Views

  • PubReader
  • Print View
  • Cite this Page
  • PDF version of this title (6.4M)

In this Page

Other titles in these collections

Recent Activity

ClearTurn OffTurn On

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...