Accuracy of Family History Reporting

Accuracy of a test (in this case reporting of family history) represents the proportion of all test results that are true (both positive and negative outcomes). If individuals reporting family history were 100 percent accurate they would correctly identify all relatives with cancer and all those without cancer. A number of metrics may be used to convey accuracy. Of these, sensitivity and specificity are not influenced by the underlying prevalence of the characteristic of interest in the population (in this case, positive family history). We therefore report sensitivity and specificity, where this is reported in (or can be calculated from) eligible papers. Consider the situation where “reporting of family history by the informant” is considered the “test”, and is compared to a “gold standard” (the real situation). In this context, sensitivity indicates how accurate informants are at identifying relatives who truly have cancer. If reporting is highly sensitive, only a few relatives with cancer will be reported as cancer-free. Conversely, if reporting is highly specific, only few relatives who are truly cancer-free are misreported as having cancer.

It is likely that accuracy of reporting will be influenced by both informant factors and factors relating to the method of capturing the family history data. As much as possible, we captured information on such attributes and considered how the results appeared to be influenced by them, although we did not attempt a formal regression analysis to examine their independent effects(s). We also examined reliability (repeatability and reproducibility) where this was possible, recognizing that this is also a product of accuracy of recall and consistency of reporting (informant factors) and performance of the instrument used to capture the data (tool factors). There are several measures of test-retest reliability such as intra-class correlation co-efficient and Cohen's kappa statistic. We note that there is no consensus on the ideal interval for assessing reliability of family history information, bearing in mind that the medical status of relatives inevitably changes over time.

As discussed in Chapter 1, three gold standards have been suggested for studies of family history taking: an “ideal” standard, a “best estimate diagnosis” (BED) standard and a “pragmatic BED” standard. We accepted the following gold standards for the presence or absence of cancer in the first and second degree relatives of the informant: (1) the relative's medical record, (2) confirmation of status by the relative's physician, (3) death certificate, (4) cancer registration, (5) direct confirmation by the relative in question. Ideally, accuracy studies should demonstrate verification of health status (presence or absence of cancer) both in relatives who are reported to have had cancer, and relatives reported not to have had cancer; however, in order to evaluate as wide a range as possible of the available literature, we did not exclude review studies which verified only the status of relatives reported to have had cancer.

We defined a priori what we meant by the degree of the relative. First degree relatives were defined as those who share one-half of their genetic information with the individual reporting family history—their full siblings, parents and children. Similarly, second degree relatives were those who shared one-quarter of their genetic information with the informant—their grandparents, grandchildren, uncles, aunts, and half-siblings.

Family History Collection Tools

We defined a FHxT as:

“A systematic and coherent approach used to capture and document family history, appropriate for the clinical setting, with the potential to lead to decision making by a clinician.”

This review focused on FHxTs which could be applied in the clinical setting, but we also included studies that described tools developed for research purposes, and for settings other than primary care, where we judged they appeared potentially applicable within primary care settings. We captured data on the following tool characteristics that may influence the clinical utility of the tool in current primary care practice.

1.

Patient targeting—“reactive” or “proactive”.

• Reactive—the tool was intended to be used only to collect family history information from individuals with perceived or recognized familial risk of cancer, including individuals concerned about cancer risk.
• Proactive—the tool was intended to be used to collect family history information from a general or targeted population coming into contact with primary care, irrespective of a known cancer risk or concern.

2.

Study setting in which the FHxT is being administered—“clinical” or “research”.

• Clinical—the primary objective of the study was to assess the use of the FHxT in routine clinical practice.
• Research—the primary objective of the study was to use the FHxT for purposes other than routine clinical practice, for example designed for data capture in epidemiological studies.

3.

Type of comparator—“best estimate” or “current practice”.

• Best estimate—the comparator was information collected by a clinical genetic specialist interview or equivalent.
• Current practice—the comparator was information collected in a way that was “standard” for the primary care setting, e.g., family history information recorded in patient charts.

Where a tool was not described as designed for or evaluated in a primary care setting, applicability was assessed by two independent reviewers against five criteria: length of tool, ease of completion, need for specialist knowledge, whether it was designed to capture data on at least all first degree relatives, and clarity of layout (including appropriate structure and logical sequence).

Risk Assessment Tools

While there is no one commonly accepted definition of a RAT, for the purposes of this study, we have followed the approach of Liu et al. who define a decision tool as:

“...an active knowledge resource that uses patient data to generate case specific advice, which supports decision making about individual patients by health professionals, the patients themselves or others concerned about them.”⁹⁷ (p90)

Defined thus, RATs have four essential characteristics:

1.

The tool is designed to aid a clinical decision by a health professional and/or patient (“user”);

2.

The tool focuses on decisions concerning individual patients (“target decision”);

3.

The tool uses patient data and knowledge from family history to generate an interpretation that aids clinical decision making (“knowledge component”);

4.

The tool is designed to be used before the health professional or patient takes the relevant decision (“timing”).

This definition encompasses a wide range of potential tool “technologies”, including computer-based decision support systems, reminder cards, guidelines, predictive scores, checklists, etc. Drawing on this definition, we have developed the following working definition of a “family history based cancer risk assessment/decision tool”, for use in this review:

“An active knowledge resource that uses family history data and other relevant evidence to generate case specific advice [knowledge component], designed to support decision making relating to management of cancer risk in individual patients [target decision component, timing component], by health professionals, the patients themselves, or others concerned about them [user component].”

We translated the four “essential characteristics” into this specific form for the context of this review:

1.

Users—health professionals, patients, members of the general population

2.

Target decision—clinical management (e.g., referral for genetic counseling), or individualized preventive management strategies (e.g., disease screening or surveillance)

3.

Knowledge component—a defined model or set of criteria which transform family history data into information which serves the target decision making process

4.

Timing—designed to be used before the health professional or patient takes the relevant decision.

The breadth of this definition potentially allows for the inclusion of a large number of guidelines, algorithms, statistical models, etc. In order to maintain the focus of this review on tools most likely to be feasible for use in primary care, we included only those which were explicitly developed for primary care, or where specialist genetics knowledge did not appear necessary to use the tool. We excluded tools where the only output was risk of carrying a cancer-associate mutation (e.g., BRCAPRO⁹⁸ or BOADICEA⁹⁹), rather than risk of disease, as we judged this required genetics specialist knowledge for interpretation. Noting also that there are many hundreds, possibly thousands, of guidelines which have been developed over the past few years around familial cancer risk, we included them only if they were part of a package, system, or intervention designed to foster their effective implementation in practice. Thus, widely used guidelines such as the modified Amsterdam criteria,¹⁰⁰ the Manchester scoring system,¹⁰¹ the UK NICE guidelines on familial breast cancer⁷² were not included unless they were part of such a system. For each tool which met the inclusion criteria, we collected data on the guideline(s) or evidence cited which appeared to form its knowledge component.

Topic Refinement

The first step during the topic assessment and refinement process was a teleconference with partner organizations. The Task Order Officer (TOO) invited topic experts and the McMaster multidisciplinary research team to define the scope of the topic to be addressed and to refine/clarify the preliminary research questions for this evidence report. An international Technical Expert Panel (TEP) was assembled to provide high level content expertise on this topic (Appendix E ^*) and to participate in conference calls on an as-needed basis throughout the data refinement and extraction phase. The TEP assisted in refining the research questions and raising methodological issues of relevance to this review.

The initial work order specified that the systematic review should be limited to adult populations and should examine the family history of at least one of the following cancers: (1) breast, (2) ovarian, (3) prostate, and (4) colorectal. The second and third questions of the review were limited to primary care settings or practitioners.

The first research question in this systematic review focuses on the accuracy of family history knowledge and reporting. The investigative team considered, but ultimately rejected, addressing this question by updating a previous systematic review.¹⁰² This review included original articles describing the accuracy of self-reported family history for breast, colon, ovarian, prostate, endometrial, and uterine cancers using verification from identified relatives' medical records, physician, death certificate, and/or verification within a population cancer registry. The limitations of this review included: lack of a delineated search strategy, overly specific search terms, non-reporting of agreement between reviewers, non-reporting of data collection forms used, and lack of clarity of reasons for excluding reports.

A number of issues relevant to the identification and evaluation of FHxTs were identified and discussed with the TEP, including: (1) the most important attributes that should be considered within each of these tools; (2) which of these elements were most relevant for primary care; and (3) the incremental value of the tool relative to current practice. The TEP recognized that the selection of gold standards for family history reporting and collection is arbitrary and that an “adequate” family history (for the purposes of making decisions relating to familial cancer risk) requires not only identifying relatives with and without the cancer, but also the relationship of the affected relative, the age of onset of cancer in those affected, and identification of several cancer types beyond the “target” cancer in question (e.g., family history of endometrial and kidney cancer is relevant in considering risk for hereditary nonpolyposis colorectal cancer).

For the purposes of the review, a definition of primary care was established with the participation of the partner at the CDC and the TEP. Primary care practitioners included family physicians/general practitioners, general internists, obstetricians, gynecologists (obstetrics and gynecology practitioners are PCPs for some women), nurses, nurse practitioners, physician assistants, nutritionists, behavior counselors.

Family history information is of clinical value only if it can be used for some form of meaningful risk stratification. Issues around risk assessment and stratification were explored with the TEP, particularly whether the various risk stratification algorithms or guidelines on which tools are based are themselves evidence-based—i.e., whether such algorithms or guidelines have adequate predictive value (i.e., clinical validity) and their use has been shown to improve patient or clinical outcomes (i.e., clinical utility). It was recognized that exploration of this would broaden the scope of the review to such an extent that it would become unmanageable. Therefore, it was determined that the validity of underlying algorithms or guidelines should be taken at face value. Thus, the focus of the review should be confined to evaluating whether tools were effective in facilitating the translation of a patient's family history information into a specific risk stratum, compared with current primary care practice, on the assumption that such stratification was worthwhile.

Search Strategy

The systematic review protocol search included the electronic databases MEDLINE^®, EMBASE^®, CINAHL^® and Cochrane Controlled Trials Register (CCTR)^® from 1990 to July 2007. In addition we retrieved and evaluated references from eligible articles. Hand searching was not undertaken for this review. However, we did review the publication types “letters” (normally excluded from reviews); the investigators suggested that, within the content area of cancer genetics, primary data information might be published as letters in some journals. We also undertook a search of relevant grey literature sources. Detailed search strategies and websites explored are listed in Appendix A.^*

Eligibility Criteria

A list of eligibility criteria was determined and standardized forms were developed in Systematic Review Software (SRS) for the purposes of this systematic review. The forms and help guides detailing the eligibility criteria can be found in Appendix B.^*

Study Selection

A team of study assistants was trained to apply the eligibility criteria in preparation for screening the title and abstract lists and the full text papers. All levels of screening were done in web-based Systematic Review Software (SRS) (TrialStat Corporation, Ottawa, Ontario Canada). Standardized forms and a training manual explaining the criteria were developed and reviewed with the screeners (Appendix B ^*). For the title and abstract phase, two reviewers evaluated each citation for eligibility. Articles were retrieved if either one of the reviewers judged it as meeting eligibility criteria or if there was insufficient information to determine eligibility. For screening of full text articles, two screeners came to consensus on the identification, selection, and abstraction of information. Disagreements that could not be resolved by consensus were resolved by one of our McMaster research team members. The level of agreement for inclusion of studies was measured using kappa statistics.

Data Extraction

Appropriate data collection forms were developed for use in the systematic review (Appendix B ^*). All eligible studies from the selection phase (full text screening) were abstracted onto a data form according to predetermined criteria. One data extractor transferred the data onto these forms, and another checked the answers for accuracy before they were entered into SRS. Data entries were verified by the investigators responsible for summarizing the different report results sections.

Quality Assessment of Included Studies. To assess the quality of primary studies, we utilized standardized rating scales with acceptable reliability and validity. The specific scale used was dependent on the study design and the research question. The Quality Assessment of Diagnostic Accuracy Studies (QUADAS)¹⁰³ was selected to evaluate studies primarily focused on accuracy (i.e., included in Q1). The Jadad scale was used for studies that were randomized controlled trials (RCTs).¹⁰⁴ For true observational study designs, the Down's and Black quality assessment scale was used.¹⁰⁵ Studies that were neither of these study designs were evaluated qualitatively without the use of formal checklists. The instruments used to evaluate quality are shown in Appendix B.^*

Summarizing Our Findings: Descriptive and Analytic Approaches

A qualitative descriptive approach was used to summarize study characteristics and outcomes. Multiple publications on the same study cohort were grouped together and treated as a single study with the most current data reported for presentation of summary results. Standardized summary tables explaining important study population and population characteristics, as well as study results, were created. Meta-analysis was not undertaken for eligible studies within this review as the clinical heterogeneity between studies was considerable.

For those papers evaluated for research Q1, where the actual numbers of true and false positive and negative results (TP, FP, TN, FN) were presented, or where enough information was given to allow us to calculate and estimate these numbers, we recalculated the sensitivities and specificities with the accompanying 95 percent confidence intervals (CI) where possible.

For those papers evaluated for research Q2, descriptive data on the attributes of FHxTs were presented. For those FHxTs that had been formally evaluated, we reported outcome data separately for those tools compared with best estimate, and those compared with current practice comparators.

For those papers evaluated for research Q3, we presented descriptive data on the attributes of RATs, including the evidence base, if any, underlying each tool. For those RATs that had been formally evaluated, we reported data on outcomes relevant to the use of the tool in supporting decisions by users in practice (e.g., the pattern of referrals from primary to specialist care, patient perceptions of their cancer risk, health professional confidence in counseling patients concerned about their risk, etc.). Data regarding the validity of the knowledge component of each RAT (e.g., the scientific basis for guidelines, the predictive value of a stratification system, etc.) were captured where possible, but it is not within the scope of this review to consider the quality of such evidence (see “Topic Refinement”, above).