Background:

Diagnostic tests inform most health care decisions, yet methods and standards used to evaluate tests focus on their accuracy and overlook additional outcomes that may be important to patients. This makes it challenging for patients and their providers to compare the full range of beneficial and harmful effects of tests. Imaging tests are one of the most frequently used kinds of tests in health care, yet there are also indications that they are overused. PCORI highlighted the lack of standards for comparative effectiveness research (CER) of diagnostic tests and the need for methods to measure the effects of diagnostic tests on patient-centered outcomes (PCOs).

Objectives:

The overall goal was to identify which PCOs are important in evaluations of imaging tests and to propose a more comprehensive and patient-centered assessment of the benefits and harms of imaging tests. We achieved this by (1) exploring patient and health care stakeholder experiences and outcomes with different types of imaging tests to identify PCOs; (2) assessing the types and frequencies of PCOs used in existing studies of imaging tests; and (3) developing consensus recommendations by which PCOs should be measured and reported in comparative effectiveness studies of imaging tests.

Methods:

A large stakeholder group including patients, caregivers, clinicians, imaging industry representatives, and methods experts was directly engaged in the research.

• Aim 1. Perform semistructured interviews with patients in primary care as well as primary care providers, radiology technologists, and radiologists to identify PCOs.
• Aim 2. Conduct a secondary analysis of systematic reviews from the American College of Radiology (ACR) Appropriateness Criteria to identify the frequency and methods used to measure PCOs.
• Aim 3. Hold a consensus meeting with stakeholders using NIH consensus methods to produce guidance defining the domains of PCOs, their placement within the testing pathway, and the factors that influence them.

Results:

We conducted 77 interviews with patients and providers, which identified several domains of PCOs important to patients and the factors that moderate them. The secondary analysis of ACR data identified 89 articles, which reported outcomes in only 10 categories. We agreed on 4 key domains of PCOs at the consensus meeting:

• Knowledge gained from the imaging test. PCOs included finding the cause of symptoms or concerns, ruling in/out a condition, inconclusive tests, false-negative/false-positive results, and incidental findings.
• Physical effects of the imaging test. PCOs included the physical impact of the test.
• Emotional effects of the imaging test. PCOs included reassurance or relief, and worry or anxiety.
• Burden of the test on the patient. PCOs included monetary costs and opportunity costs (ie, the loss or benefit from other alternatives when one alternative is chosen).

Conclusions:

Our study defined PCOs of imaging tests, and it directly addresses a current methodological gap in CER of diagnostic tests. The PCOs of diagnostic tests fall into 4 domains. Patients should have information on the PCOs that occur within each domain to inform choices about the use and selection of imaging testing. CER of imaging tests needs to measure and report PCOs within these domains, and this could be implemented as part of shared decision-making with patients.

Limitations:

The patients and providers selected may not be representative of all clinical settings. Our sample did not achieve a balance of sex, racial, and ethnic diversity. The secondary analysis yielded only limited data on PCOs, yet it reflects the paucity of data currently used to inform decision-making. Our findings may not be generalizable to all types of diagnostic testing.

Methodological Gaps Addressed by the Research Study

A major aim of the PROD study was to address the gaps in methodology pertaining to prioritizing, measuring, reporting, and communicating PCOs related to diagnostic tests. Although initial conceptual work^7-9 describes possible types of PCOs for tests, there is a need for definitive research to produce guidance for test developers, the testing industry, diagnostic test researchers, patients, caregivers, and health care professionals, in multiple areas related to PCOs of diagnostic tests.

A report conducted for PCORI¹⁰ on the design, conduct, and evaluation of diagnostic testing recommended minimum standards for all CER studies of diagnostic tests. It concluded that the findings of CER studies “should be presented in ways that are accessible to patients and the broad range of stakeholders and should address PCOs.” This report also highlighted the importance of measuring the effects of tests on a patient's well-being and recommended that “those planning diagnostic test studies should obtain an understanding of the nature and frequency of expected patient-relevant outcomes of testing.”¹⁰ The report also pointed out, however, that CER studies of diagnostic tests are not currently designed to take into account the clinical pathways involving the tests or the implications of tests on the process of care and patient outcomes.

These findings were further highlighted in the 2012 PCORI Methodology Report,¹⁰ which noted that

diagnostic testing's impact on patient outcomes has been traditionally understudied in clinical research. Studies of diagnostic tests tend not to identify all of the pertinent effects on patients, particularly long-term benefits and harms, as well as cognitive, emotional, social, and behavioral effects.

The report goes on to state:

Although these guidelines address the reporting of diagnostic or predictive accuracy studies, standards have not been established for studying the impact of diagnostic tests on subsequent care or patient outcomes.

Other researchers have also described the need to balance patient-important outcomes resulting from true and false positives and negatives with test complications (eg, pain, anxiety) in deciding whether to recommend a test. They conclude that

the application of the approach requires a shift in clinicians' thinking to clearly recognize that, whatever their accuracy, diagnostic tests are of value only if they result in improved outcomes for patients.¹¹

Overall Goal

We conceived the research study in response to PCORI's request for proposals to address this methodological gap. We recruited stakeholders to aid in planning the overall study and to set an overall goal and specific aims to meet this goal.

The overall aim of the PROD study was to develop a consensus-based, multistakeholder guideline and recommendation for reporting and presentation of PCOs of diagnostic imaging tests to facilitate more informed decision-making regarding these tests.

The PROD research is expected to have significance for patients, researchers, diagnostic test developers, professional bodies, and test regulatory bodies. Our findings filled the research gaps identified by the PCORI Methodology Report on standards for studies of diagnostic tests. We anticipate that the study findings will help ensure that researchers include PCOs when planning CER studies of diagnostic tests. This may lead to guidance that will make it easier for clinicians, patients, and caregivers to compare the potential benefits of an imaging test (eg, correct diagnosis, reassurance, treatment targeting) with potential harms of that test (eg, radiation exposure, invasiveness, false-positive results, incidental findings, anxiety, out-of-pocket expenses).

The proposed consensus guidelines produced from the PROD study will also potentially inform diagnostic test reporting guidelines, such as the Standards for Reporting Diagnostic Accuracy Studies (STARD),¹² that currently are used for diagnostic study reporting. The explicit inclusion of PCOs in evaluations of imaging tests would be expected to advance research and reporting in this area to facilitate a far more detailed and comprehensive understanding of the full impact of diagnostic tests on patients. This could therefore provide new tools and guidelines that PCORI and other stakeholders can use for comparative evaluations of new and existing diagnostic tests, as well as for communication of these results to patients.

Overall Approach

The overall PROD research study approach was modeled on strategies successfully used to develop guidelines for the reporting and quality assessment of research studies. This included methods used to develop the Grading of Recommendations, Assessment, Development and Evaluation (GRADE) guidelines,¹³ methods used to develop the standards for reporting of research studies (EQUATOR Network),¹⁴ and methods used to develop the effectiveness guidance recommendations for the incorporation of patient-reported outcomes in oncology studies.¹⁵

The multistep process, which was modified from GRADE and EQUATOR Network procedures specifically, involved the following steps: (1) identify the clinical questions to address; (2) consider all outcomes important to patients in conjunction with our patient-researchers; (3) conduct systematic reviews for each PCO; (4) present the results of the review using the GRADE evidence profile; (5) consider balances of quality, values and preferences, and benefits and harms; (6) draft recommendations; and (7) finalize the guideline following feedback from multiple stakeholders, publication, and wide dissemination. This process was modified, however, as noted in subsequent sections, to provide a more suitable approach to achieving the key study outcome of developing consensus for new methods.

We engaged stakeholders during the conception of the study proposal, and we recruited and met quarterly with the PROD study stakeholder advisory board. We approached the overall goal of the PROD study using 3 aims (Figure 1).

Figure 1

Overall Plan for the PROD Study.

Background

The PROD study developed an engagement plan to meet PCORI Standards Associated with Patient-Centeredness and to inform the design, conduct, interpretation, and dissemination of the planned research.¹⁸

When the proposal was initiated in early 2014, we engaged informally with a group of 5 patients to provide initial input into the planned project and overall strategy. We met individually with these patients (aged 56-72 years; 2 male, 3 female) who had experience with imaging tests. We also asked them to complete a brief questionnaire on their experiences with imaging tests and their opinions about the need for research on the effects of these tests on PCOs. There was strong agreement among patient stakeholders that patients should be made aware of the positive and negative effects of diagnostic tests, and they were enthusiastic about the proposed study. This initial engagement encouraged us to pursue the proposed research further and to engage with a wider group of stakeholders as the project became formalized.

We began by partnering with a patient and consumer health advocate to serve as a core (and funded) research team member. With extensive experience communicating medical issues to consumer and lay audiences, this stakeholder helped craft the research questions and approach described in the study proposal, participated in all investigator meetings, and advised research decisions from the patient perspective; this stakeholder is listed as a co-investigator on study outputs.

Feedback on the proposal was obtained from University of Washington (UW) Centers for Comparative and Health Systems Effectiveness (CHASE) Alliance researchers; Washington, Wyoming, Alaska, Montana, and Idaho (WWAMI) primary care providers (PCPs); the ACR; the American Board of Internal Medicine Choosing Wisely initiative; 2 major imaging multinational companies (Phillips and GE Healthcare); and 1 smaller handheld ultrasound device manufacturer. Focus groups were conducted with PCPs at 2 WWAMI-region Practice and Research Network (WPRN) sites in which clinicians discussed point-of-care tests in general and their challenges in knowing the pros and cons of such tests. This input helped us refine the specific aims and revise the study methods.

We created a stakeholder advisory board using the 7Ps Framework¹⁹ to identify stakeholders in PCOR and CER and a second framework from Deverka et al²⁰ to target stakeholder groups that would be responsible for or affected by the health decisions related to our research question. The final list of stakeholder groups recruited is shown in Figure 2.

Figure 2

Stakeholder Advisory Board Framework.

Patient stakeholders were identified from the Comparative Effectiveness Research Translation Network (CERTAIN) Patient Advisory Network of >200 patients, caregivers, and researchers interested in collaborating across diverse research topics. PCPs were recruited from the 5-state WPRN, which provided a diversity of geography and practice types and enabled us to reach rural and underserved populations. Radiology stakeholders were recruited through our relationship with the ACR and the Department of Radiology at the UW Medical Center. Diagnostic methods experts were identified as leading researchers internationally based on key authors in this area of research methods related to diagnostic testing. Imaging industry, ACR, and health care nonprofit stakeholders were recruited through professional contacts and networking of the principal investigators (PIs).

In total, 26 stakeholders were recruited to serve on the PROD study stakeholder advisory board. There were 8 patient advocates, 4 PCPs, 1 radiology technologist (in addition to 1 radiologist on the core research team), 5 diagnostic methods experts from 4 countries (United States, United Kingdom, The Netherlands, Australia), 4 imaging industry representatives, 3 senior staff from the ACR, and 1 stakeholder from a health care nonprofit organization. The stakeholders varied in age, geographic location (global), and background. The PCPs, radiologist, and 2 of the methods experts were MDs; the remaining stakeholders were not.

Stakeholder Responsibilities and Participation

Adopting the Six-Stage Model for Patient-Centered Outcomes Research and Comparative Effectiveness Research,^19,20 we engaged stakeholders in evidence prioritization and evidence generation (within aim 1 of the PROD study), evidence synthesis (aim 2; evidence interpretation and integration, dissemination, and application), and feedback and assessment (aim 3).²¹ PROD study stakeholder advisory board members were asked to do the following:

• Participate in 2 to 3 annual advisory board conference calls
• Participate in 1 to 2 annual strategy or smaller “quality-time” conference calls with the research team
• Advise on methods used during the qualitative data collection
• Interpret results on patient outcomes that arose from the qualitative interviews
• Advise on the secondary systematic review methods
• Review the secondary systematic review synthesis of articles and findings
• Attend a 2-day advisory group consensus meeting in Seattle, Washington, to generate clear guidelines for the diagnostic testing community and provide expert advice at a symposium
• Advise on communicating findings and guidelines to consumer, expert, and lay audiences
• Aid in the planning and actual dissemination of information to the community

Midway through the study period, a formal publication policy was circulated to the research team and the stakeholder advisory board. It outlined the requirements to meet authorship on scientific publication manuscripts and conference presentations based on International Committee of Medical Journal Editors recommendations, as well as a list of anticipated study outputs and a general workflow schedule for outputs.²²

Stakeholder Advisory Board Participation

At the onset of the study, stakeholders were asked to participate in quarterly conference calls with the entire board. After the initial few stakeholder advisory board meetings, study PIs decided that a more targeted meeting structure would better serve the study needs. The meeting structure therefore evolved to 2 overall advisory board meetings per year and 2 to 3 targeted quality-time meetings with specific stakeholder groups, depending on study needs at the time.

For example, during data collection and analysis of the patient interviews, 2 quality-time meetings were scheduled specifically with PROD study patient stakeholders to advise on interview guides and data interpretation. Every meeting was conducted either by conference call or video conference. The meetings were scheduled 2 to 3 months in advance to maximize attendance, and meeting materials (such as preliminary results or draft manuscripts) were circulated 1 to 2 weeks before the meeting. During the meeting, the PI briefly presented these findings and solicited feedback from each stakeholder. Most stakeholders participated actively in the calls; only when a particular person was silent or particular feedback was needed were stakeholders specifically asked for their opinion. The meeting minutes were circulated after each conference call.

A smaller group of the stakeholder advisory board was assembled to provide specific feedback for aim 2 of the PROD study. We asked board members to volunteer for the technical expert panel (TEP) based on expertise and/or interest in participating more directly during this step of the research. We used the TEP to guide decisions such as search strategy and inclusion criteria of the literature review, as well as for definitions and grouping of PCOs in the study findings. The TEP structure helped us obtain timely feedback between our semiannual stakeholder meetings.

Toward the end of the research project, stakeholders were invited to participate in a 2-day, in-person meeting in Seattle to review all of the study data generated and develop a consensus statement. Reimbursement was provided for participants to attend this meeting. Participants who could not travel participated by teleconference. All study manuscripts, meeting agenda/materials, and background literature, organized by meeting session, were provided 2 months before the meeting.

The meeting followed the NIH consensus methods to generate input and feedback for the proposed PCO domains and framework recommendations.^23,24 A total of 4 sessions over 2 days began with session background and talks from specific stakeholders. In these sessions, a question was posed and the participants broke into 4 smaller groups to discuss it. The small groups presented their discussion to the whole group to build consensus based on similarities across all small-group discussions. The meeting was audio recorded and transcribed. Final meeting notes were circulated to the stakeholder advisory board.

PROD study stakeholders have participated as co-authors on 6 drafted or published scientific manuscripts and 10 conference or meeting presentations to date. Authorship has spanned multiple stakeholder groups per manuscript, with 16 of the 26 stakeholders participating at least once as a co-author. Each study output also acknowledges the overall PROD study team, which includes the core research group and the stakeholder advisory board.

Evaluation of Stakeholder Engagement

Three levels of patient involvement in research have been described: consultation, collaboration, and control.²⁵ The PROD project extended this framework to a wider group of stakeholders, who were involved at these 3 levels:

1. Stakeholders were consulted to inform research activities. Informal patient meetings, input from our patient consumer advocate, focus groups with PCPs, and meetings with several groups involved in diagnostic research all helped develop our overall proposal. As we conducted qualitative data collection, each stakeholder group was given an opportunity to shape the interview guides and to contribute to data interpretation. Stakeholders who participated in the TEP group guided the development of the protocol for the secondary review of the ACR AC literature. The TEP met several times during the planning phase to discuss search procedures, eligibility criteria, and definition of PCOs.
2. Collaboration or active and ongoing partnership with stakeholders during research was achieved through the purposeful meeting schedule. Early dissemination of the agenda and other materials equipped stakeholders to plan for and participate actively in meetings. During the 2-day in-person meeting, all stakeholders were invited to give 5- to 10-minute presentations reflecting their expertise, providing another opportunity for them to contribute to the scientific discussions. Last, all stakeholders were invited to collaborate on scientific outputs as co-authors through data analysis and interpretation of study findings. Stakeholders who volunteered were listed as co-authors on publications and worked with the primary author on manuscript editing throughout the publication process.
3. Control or user-controlled research, where the locus of power, initiative, and subsequent decision-making is with the stakeholders, was not followed exactly with the PROD study. Although stakeholders had significant power in several areas of the research, such as the development of interview guides or the creation of the TEP group to execute aim 2, their main decision-making influence occurred in aim 3 in their participation in the final consensus statement. Core researchers viewed the stakeholder advisory board as part of a team rather than being controlled by a particular stakeholder group.

We included stakeholders from most groups listed in the 7Ps framework presented by Concannon et al.¹⁹ Although no formal evaluation was conducted to measure the success of the PROD study stakeholder advisory board, some metrics and anecdotes suggest that engagement was successful.²⁶ The first metric, attendance at the advisory board meetings, shows that an average of 72% (range, 40%-92%) of the PROD study stakeholders attended stakeholder meetings. These numbers are most likely underestimated, as some people who joined conference calls late were not always included in the attendance numbers. Eighty-two percent of the stakeholder advisory board attended the 2-day meeting either in person or via the conference line (used by 2 attendees). Most stakeholders traveled from out of town to attend the meeting.

Our stakeholder engagement was continuous from the study conception to conclusion, and the board grew in size as study needs evolved. The quality-time meetings set up early in the PROD study were a novel and highly effective way to pinpoint expertise among stakeholders and facilitate deeper engagement during crucial points of the study. We recommend this approach to any project that has a stakeholder board with more than 10 to 12 people. Although anecdotal, we received positive feedback from many stakeholders at the in-person meeting. One stakeholder commented that the study was so well run that they stayed engaged through the entire study instead of losing interest and dropping off the board. Another appreciated how well organized the study was and how well the meetings were laid out. Last, as the core research team reflects on the value of the stakeholder advisory board, a major finding we experienced was a consistent refocusing of discussions on the patient experience when meetings diverged from the agenda. Including stakeholders from groups other than patients, providers, and researchers also helped attain additional perspectives related to the study. For example, given that industry uses research (including patient-centered research) to develop new devices, understanding industry needs and constraints in their development process helped the project frame and disseminate research findings. Stakeholders also pushed the core research team to improve research methods, such as recruiting broader diversity in our research participants. We fell short of this goal but appreciate the recommendations and reminders to continually improve the rigor of the research by incorporating as many perspectives as possible.

Our stakeholder engagement plan had some limitations. First, we did not execute a formal evaluation to measure stakeholders' level of engagement in the project, nor has the core research group evaluated the effectiveness of the stakeholder advisory board.²⁶ Second, we did not create or disseminate a formal decision-making process for the study.²⁷ We felt the board operated organically, with the final decision resting with the PI. A formal policy included in the stakeholder responsibilities and role, however, might improve transparency of the expectations for all members of the team. Third, the disparate geographical locations of stakeholder advisory board members was challenging. With members located across the world, phone/video conference was effective, but the diversity of time zones meant we could not feasibly include everyone at a reasonable time. To overcome this, we scheduled times that were convenient for most of the board. Circulating detailed meeting minutes and asking for specific feedback from members who could not attend kept them engaged in the study, as did one-on-one telephone meetings with our stakeholder in Australia.

We feel that our engagement plan provides a successful example of incorporating a diverse body of stakeholders into a methodology-based research project, and one that may be applicable to other research groups. A key takeaway we would change in future projects would be to plan an annual in-person meeting. The stakeholders who attended the 2-day consensus meeting were highly valuable in shaping the direction of the final guidelines and conclusions of the PROD study, an outcome that may not have happened if we had not met in person. The core research team felt that a similar meeting would have been useful after the completion of each study aim. A novel addition to typical stakeholder engagement methods is the inclusion of the quality-time meetings in the meeting structure. We suggest that other researchers include this type of meeting if their stakeholder groups are large. Engaging stakeholders from conception through every phase was a strength of the PROD study, because it ensured this project was meaningful to a broad range of stakeholders and would measure exactly what we needed to meet stakeholder expectations. As study conclusions become finalized, we plan to use stakeholders to participate in the dissemination of study findings to their respective stakeholder groups.

Summary

Stakeholder engagement was instrumental to the PROD research program and involved a deliberate effort to engage with a wide range of stakeholders starting with study conception and continuing for the duration of the 3-year research program. The 26 stakeholders who were recruited to serve on the PROD study stakeholder advisory board included patient advocates, PCPs, a radiology technologist, a diagnostic methods experts, imaging industry representatives, the ACR, and a health care nonprofit organization. The core research team engaged with the board through regular meetings, including advisory board meetings, quality-time meetings with smaller numbers of stakeholders, use of a technical expert panel, and phone/teleconference calls with individuals. This engagement shaped the research conducted, the study outcomes, and, in particular, the process used to guide the creation of the consensus recommendations (see the “PROD Consensus Statement and Recommendations” section).

Background

To carry out aim 1, we sought to identify PCOs from the perspectives of patients and their health care team. Extensive qualitative research has already been undertaken for some aspects of PCOs related to imaging, primarily focused on the harms from screening, in particular, mammography.²⁸ We therefore decided to explore PCOs across a range of patients involving different imaging modalities. We also sought the perspectives of their health care team, namely, their PCP and the radiology team (ie, radiologists and radiology technologists).

We focused our research on patients and clinicians in primary care, as imaging tests fulfill multiple roles in this setting, including screening, diagnosis, and monitoring of disease. Indeed, imaging tests are ordered or conducted in approximately 14% of office visits in the United States.²⁹ In addition, concerns about the potential overuse of imaging tests are growing, with almost 75% of physicians identifying unnecessary testing as a serious problem.⁵ Some of the high use of testing is driven by overestimation of test benefits, diagnostic uncertainty, medicolegal concerns, and patient requests.³⁰ Radiologists have also identified the need for more detailed or nuanced methods to more fully evaluate the outcomes of imaging procedures.^31,32 This is partly in response to growing interest among the radiology profession for a more patient-centered approach to radiologic testing.

The goal of the research conducted within this section was to explore patient experiences, PCP perceptions of patient experiences, and radiology provider perceptions of patient experiences and the subsequent outcomes related to imaging testing. We also sought to explore factors that might influence these outcomes.

Methods

We conducted semistructured interviews of 3 participant groups to qualitatively explore the outcomes of imaging tests through experiences that patients and providers (primary care and radiology) noticed.³³ The interviews were used to determine themes related to outcomes that may be important to patients.

Participants were recruited from clinics in Washington and Idaho through the WPRN. We recruited participants from 3 groups: (1) adult patients from 4 primary care clinics who had had an x-ray, CT scan, MRI, or ultrasound in the previous 12 months; (2) PCPs from those same clinics who were either family physicians, internists, family medicine nurse practitioners, or family medicine physician assistants; and (3) radiology providers who were either x-ray, ultrasound, CT scan, or MRI technologists or general, neurologic, body, or breast radiologists. Patients were purposively sampled based on the type and timing of their last imaging test. We used convenience sampling to recruit primary care and radiology providers through flyers circulated at staff meetings or through email. The UW Human Subjects Division approved the study. All participants provided verbal consent for participation.

Separate interview guides were developed for each participant group to direct the interviews. The patient guide was informed by several frameworks that propose primary domains of PCOs, as well as the time points on a patient's imaging journey (before, during, or after) where these may occur.^34-36 Patients and patient advocate stakeholders from the PROD study stakeholder advisory board reviewed the interview guide for relevancy. The patient interviews informed the content for the PCP and radiology interview guides. Interviews were conducted from September 2016 to December 2017 by phone or in person by 1 of 4 trained interviewers on the research team. The interviews were audio recorded and transcribed. Transcriptions were checked for accuracy against the audio recordings. The interviews were conducted until data saturation was met for each participant group.³⁷

The patient, PCP, and radiology transcripts were analyzed separately.³⁸ Two research members immersed themselves in the transcripts, open-coded the initial transcripts, and compared codes to develop a coding framework for each participant group.^39,40 The coding framework was applied to the remaining texts and revised iteratively until the research team agreed on a final framework. This process was repeated for patients, PCPs, and radiology providers.

Thematic analysis guided the patient interviews and radiology interviews,^41,42 whereas classic content analysis methods were used to interpret PCP interviews.^43,44 We used Dedoose software (version 7.0.23; SocioCultural Research Consultants, LLC) to assist with the qualitative analysis. The codes from each coding session were reviewed for themes and subthemes based on the analytical method employed. Final themes of PCOs or perceived patient outcomes and sample quotes are presented in the “Results” section below.

Results

A total of 77 patients and providers participated in a single semistructured interview. Tables 1a to 1c summarize the demographics of the 45 patients, 16 PCPs, and 16 radiology providers who participated in the interviews.

Table 1a

Characteristics of Patients Who Were Interviewed.

Table 1b

Characteristics of Radiologists and Radiologic Technologists Who Were Interviewed.

Table 1c

Characteristics of PCPs Who Were Interviewed.

Analysis of the patient interviews established 4 central domains of PCOs: (1) knowledge gained from the imaging test, 2) its contribution to the patient's overall health care journey, 3) physical experiences during the test procedure, and 4) impacts of the testing process on emotions. We identified example outcomes in each domain.

Analysis of the PCP interviews suggested an additional outcome domain related to the cost of testing, which had not been specifically raised by patients in our data gathering. Our analysis also led us to identify many factors that influence PCOs. The definition of these factors evolved throughout the study, but the research team identified these influencing factors as moderators to PCOs. We defined a moderator as a variable that specifies when certain effects hold, such as the direction or the strength of a relationship between the predictor (in this case, imaging testing) and the PCO.¹⁷

The radiology provider interview analysis confirmed 3 of the patient outcomes and expanded the outcome of costs to encompass additional burdens (such as time off work), leading to a domain of patient burden. On further analysis of the moderators of patient outcomes, we discovered that many of these moderators influence PCOs throughout the imaging test process.

Discussion

We aimed to understand patient experiences from the perspectives of the patient, PCP, and radiology provider. Our qualitative analyses identified 4 primary domains of PCOs resulting from those experiences: emotional, physical, knowledge, and contributions or burdens of the health care journey. The patient interviews primarily helped define the domains and outcomes that fall into those domains, and the provider interviews confirmed these domains, added new concepts to 1 domain, and aided in identifying many moderators that determine when certain effects of a domain hold. For example, the level of communication between provider and patient is a moderator; limited communication may make a patient feel anxious and uncertain about what is going to happen, thus impacting the domains of emotion and knowledge.

The findings from the 3 qualitative studies suggest a range of risks and benefits of testing far beyond accuracy. Even the domain of knowledge, which is most closely aligned to diagnostic accuracy in that most patients are driven by the desire for a diagnosis, diverges from accuracy in some respects. For example, the knowledge gained from an imaging test, while accurate, may not lead to a definitive diagnosis from the patient's perspective, which might be their most important outcome. We hypothesized that there would be outcomes that patients might consider important while remaining unaware of all the risks and benefits of imaging testing. This was confirmed by our addition of patient burden to the health care journey. Both provider groups recognized outcomes in this domain, while patients placed more weight on having a diagnosis and moving forward with health care than on any negative effects that may come with imaging tests.

Our research therefore provides a novel investigation of PCOs of imaging tests in primary care. This study is strengthened by triangulation of the perspectives of patients, PCPs, radiologists, and radiologic technologists. Through triangulation, we were able to confirm outcome domains and identify newer domains not recognized by certain perspectives. Given the wide scope of the PROD study, we inevitably attempted to balance depth (exploring PCOs in sufficient detail to be credible) with breadth (conducting research across a potentially vast range of patient characteristics, imaging modalities, clinical situations, and health care settings). We acknowledge that because our sample was focused on primary care settings, the outcomes we identified may apply differently for other patient care settings and populations, particularly those in high-acuity settings or with significant current illness. Our study findings may be limited because interviews were conducted among English speakers in the northwestern United States through either purposive or convenience sampling. Some perspectives may have been missed and will require follow-up research to capture perspectives from more diverse patient populations. Although we attempted to capture patient data as close to the imaging test date as possible and asked providers to recount general observations, the data may suffer from recall bias. While the points of view of the researchers may have influenced the interpretations of findings, the researchers worked closely with a stakeholder advisory board (which included patients, PCPs, radiologists, and radiologic technologists) to interpret the findings and draft manuscripts.

To further explore the breadth and depth of PCOs that are important to a range of patients across imaging modalities and clinical scenarios (including settings beyond primary care), we conducted a scoping review of qualitative research that has described patients' experiences of imaging testing. This review has been submitted for publication in a peer-reviewed journal and is briefly described here.¹⁶ The scoping review followed the Arksey and O'Malley methodological framework for conducting scoping reviews⁴⁵ and drew on more recent recommendations for enhancing the methodological rigor of scoping reviews from Levac et al⁴⁶ and Colquhoun et al.⁴⁷ The key steps in the scoping review were (1) identifying the research question(s), which were “What are patients' emotional, knowledge, and physical preferences in relation to imaging tests either before, during, or after imaging procedures?” and “Why are these preferences important to patients”; (2) identifying relevant studies using a defined search strategy, using search terms mapped onto the relevant domains of the SPIDER (Sample, Phenomenon of Interest, Design, Evaluation, Research type) framework for qualitative evidence synthesis to inform bibliographic database searching; (3) selecting studies that reported PCOs of imaging studies; (4) charting of the abstracted data; and (5) collating, summarizing, and reporting the results.

A total of 25 articles were included in the scoping review, with data on 656 patients and 23 caregivers covering a range of imaging modalities, target conditions, and populations. The included studies largely focused on secondary-care settings rather than on primary care.

The PCOs that we identified were categorized into domains of information or knowledge gained from a test, the emotional impact of the test, and effects on physical symptoms that patients may experience during or after the test. Patients identified many different outcomes within each of these 3 domains, some of which can broadly be considered positive (beneficial) and others negative (harmful). The scoping review did not find as much evidence of the burden of testing on patients, although we are aware of literature on the issue of waiting times to receive an imaging test and its impact on emotions and life planning, as well as the issue of the burden of loss of control that some patients experience.^48,49 Other outcomes reported in the literature were not identified from our interviews, such as the impact of testing on behavioral or social outcomes.^7,48-51 This may be because these outcomes may be more distal or that the evidence for them is sparse. The scoping review also built on findings from our primary qualitative research regarding factors that can be considered moderators and mediators of the test experience. Moderating factors are conditions that influence an outcome by indicating conditions when that outcome will hold, whereas mediating factors explain how or why this relationship might exist.¹⁷

Summary

The main conclusions from this section are that our primary qualitative research to understand patient experiences and outcomes from testing from the perspectives of the patient, PCP, and radiology provider identified 4 primary domains of PCOs resulting from those experiences: emotional, physical, knowledge, and burden. A scoping review of the qualitative literature provided further evidence from a wider body of published literature for PCOs in each of these domains, with the exception of the burden of the test or testing process. An additional conclusion from both the primary qualitative studies and scoping review was that factors from the patient, provider/health care system, and clinical situation can moderate outcomes, which can be explained by various mediating factors.

Background

While several groups within the radiology specialty have identified the need to include PCOs within their field, the extent to which PCOs are currently used as part of decision-making by radiologists and referring providers is unclear. Indeed, understanding patient-centered domains was identified by the Society of Interventional Radiology as a priority within the area of interventional procedures.⁵² The Association of University Radiologists Radiology Research Alliance Task Force identified the assessment of prevention, diagnosis, and treatment options, as well as the communication and dissemination of research, as PCO priority research areas.⁵³ The ACR Patient and Family-Centered Care initiative⁵⁴ has also highlighted the importance of effective communication to give patients the information necessary to make informed choices.⁵⁵

The aim of this component of the PROD study was to identify the frequency and type of PCOs reported in studies of diagnostic imaging testing. This was expected to lead to evidence that could provide data on which PCOs are currently measured and reported and which are absent. It also aimed to identify the tools or measures that have been used to measure PCOs.

Methods

Note: Some of the text in this “Methods” section is adapted from Zigman Suchsland M, Cruz MJ, Hardy V, et al. Qualitative study to explore radiologist and radiologic technologist perceptions of outcomes patients experience during imaging in the USA. BMJ Open. 2019;10(7):e033961. doi:10.1136/bmjopen-2019-033961 [PMC free article: PMC7375501] [PubMed: 32690729] [CrossRef]

A protocol was developed in conjunction with a TEP, which convened to oversee the conduct of this study. The panel consisted of individuals with expertise in radiology and systematic reviews, imaging industry representatives, members of the ACR, diagnostic methods experts, and patients/patient advocates. The design and reporting of this secondary analysis was informed by PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-analyses) guidelines.⁵⁶

The ACR systematically reviews published literature to provide evidence to inform expert and consensus-based recommendations. The AC are used by referring physicians, radiologists, and other providers to guide imaging and treatment decisions across multiple clinical areas. Within each clinical area, specific topics are identified with accompanying evidence tables that list studies identified through systematic searches. Each evidence table summarizes the study title and type, sample size, study objectives, and main findings of identified articles; the tables can be found at https://acsearch.acr.org/list.

We included evidence tables for topics within clinical areas relevant to the PROD study—specifically, topics of screening, diagnosis, surveillance/monitoring, and staging imaging on adult patients (breast, cardiac, gastrointestinal, musculoskeletal, neurologic, thoracic, urologic, vascular, and women's health). We excluded pediatrics, radiation oncology, and interventional radiology topics. Primary studies cited in eligible evidence tables were included if they reported PCOs from an imaging test used for any purpose. The included studies were conducted in any clinical setting (eg, hospital, ambulatory) in high-income countries and are reported in English. Clinical topics were excluded if a search strategy was not available on the ACR website. We excluded quantitative studies with <20 patients as well as technical feasibility studies, clinical guidelines, systematic reviews, letters, and abstracts where the full-text article was not available. Studies solely reporting any of the following outcomes were also excluded: diagnostic accuracy, incidence or prevalence, epidemiological associations, sonographic parameters, and anatomic or pathologic outcomes.

There is no widely accepted definition of PCOs specific to diagnostic testing, but we adopted PCORI's definition,⁶ further influenced by Bossuyt and McCaffery's framework.⁷ We excluded outcomes that we considered to be downstream consequences due to changes in clinical management from the test. Because research-specifying outcomes that are important to patients for imaging tests are limited, we did not have an exhaustive predefined list of PCOs for inclusion, meaning that some PCOs were considered and discussed on a case-by-case basis. However, PCOs identified a priori based on previous research included psychosocial, quality of life (QOL), incidental findings, downstream testing, test-related complications, and time (eg, test preparation, testing procedure, or time burden to patients), as well as physical effects of the test or testing process.^7,35,38,49 We included studies where PCOs meeting these criteria were included, regardless of whether they were solicited from patients themselves or were primary or secondary outcomes of the study.

We systematically reviewed the clinical areas listed on the ACR AC website (https://acsearch.acr.org/list) to identify relevant topics within each clinical area and excluded any that were not relevant to the PROD study or that lacked an accompanying search strategy. Evidence tables from eligible topics were downloaded and manually searched for potentially eligible articles. Full details of the ACR search strategies (including search terms, search dates, selection criteria, and bibliographic databases searched) are available on the ACR website (www.acr.org/ac). We used evidence tables from the most recent searches performed by the ACR (range, March-December 2017) and last accessed the ACR website AC to corroborate the included topics on January 10, 2018.

The article summaries reported within each evidence table were screened for eligibility by at least 2 authors. Full texts of articles identified as potentially relevant were retrieved and reviewed using inclusion/exclusion criteria. Final inclusion was determined by consensus between 2 authors, with discrepancies resolved by a third. The primary reason for exclusion (eg, ineligible clinical area, design, population, outcome, not an imaging test) was recorded, and duplicate articles (ie, those appearing in >1 clinical area) were identified and removed.

Two authors extracted (not independently) data regarding study year, design, clinical topic, imaging modality, and imaging purpose from the studies, as well as details of the PCO, including its definition, and the measurement method from each included article to an Excel spreadsheet. The extracted data were checked for accuracy by a third author.

Each article had a study quality rating assessed by the ACR according to their methodology based on elements from the GRADE methodology.⁵⁷ As our objective was to identify PCOs rather than to synthesize data to make recommendations to inform practice, we did not repeat the quality appraisal.

Data were analyzed using a descriptive synthesis. The identified PCOs were grouped into categories by the 2 authors. We present our descriptive synthesis by PCO, stratified by imaging modality.

Results

Note: Some of the text in this “Results” section is adapted from Zigman Suchsland M, Cruz MJ, Hardy V, et al. Qualitative study to explore radiologist and radiologic technologist perceptions of outcomes patients experience during imaging in the USA. BMJ Open. 2019;10(7):e033961. doi:10.1136/bmjopen-2019-033961 [PMC free article: PMC7375501] [PubMed: 32690729] [CrossRef]

We reviewed the 254 clinical topics listed on the ACR website as of December 30, 2017. After excluding 92 topics not relevant to the PROD study (ie, identified by the ACR as “interventional,” “pediatric,” or “radiology oncology”), 162 topics remained. Of these, 68 had no accompanying search strategy available at the time of the review, and evidence tables for 11 topics appeared in 2 clinical areas. Evidence tables for the remaining 83 clinical areas were screened, another 14 of which were excluded due to no eligible articles (Figure 3).

Figure 3

Flowchart of the ACR Search Process.

Our review consisted of 89 eligible clinical topics/evidence tables, which included 5196 article summaries, of which 4815 were excluded. Of the 381 potentially eligible full-text articles, 292 were excluded because of ineligible clinical areas (n = 29), study design (n = 57), population (n = 9), outcome (n = 156), intervention not being an imaging test (n = 12), for a combination of these reasons (n = 15), being a duplicated article (n = 8), or because there was no full text available (n = 6). A total of 89 articles were included in the descriptive synthesis.

Characteristics of PCOs

The most frequent PCOs identified were radiation exposure (n = 37), downstream testing (n = 20), complications (n = 19), and indeterminate or incidental findings (n = 10). Additional PCOs identified included QOL (n = 7), physical discomfort (n = 5), patient values and experiences (n = 4), patient financial and time costs (n = 4), psychosocial outcomes (eg, depression, anxiety, self-efficacy, claustrophobia) (n = 4), and test duration (n = 2) (Table 2).

Table 2

PCOs Reported in the Included Studies.

Radiation exposure was most commonly reported for CT imaging (36/37 studies). These reported ionizing radiation risk to specified anatomical regions, or the whole body (expressed as effective radiation dose), from an individual imaging test or series of tests. Seven articles aggregated doses to determine cumulative radiation exposure, but only 3 of these noted the duration of follow-up. Radiation exposure was often estimated rather than measured directly. Exposure estimations used combinations of parameters that varied by study (eg, sex, anatomic area, scanning range). Parameters were entered into, or derived from, dosing formulas using software (eg, XDose) or preset scanning protocols/formulae (eg, Monte Carlo simulations for anthropomorphic phantoms).

Downstream testing, namely, additional imaging following an initial test, was most often noted for mammography (9/20 studies) and CT scans (8/20). Four of the mammography studies used reporting system categories (eg, the Breast Imaging Reporting and Data System). Seven studies of CT scans used medical record review to identify subsequent tests or procedures performed at the initial assessment or during follow-up, although only 3 articles defined the assessment/follow-up period. The need for further testing was based on expert interpretation of incidental findings identified or by the number of subsequent imaging tests conducted until a diagnosis was reached.

Complications were most frequently identified for CT scans (13/19 studies) and MRI (7/19) and included adverse events or reactions to contrast material (eg, nephrotoxicity). In all 19 articles, these events were identified from clinical observation within a defined time frame based on changes in laboratory tests and/or medical record review, and 3 of these articles also included patient reports (eg, report of procedure discomfort).

Indeterminate and incidental findings were found only in studies of CT imaging (10/10 studies) and referred to additional findings of varying (ie, malignant or benign) or unspecified clinical importance that indicated immediate or deferred investigations/treatment. These outcomes were determined through expert medical note review to identify the presence and histology of indeterminate and incidental findings or to classify them using established grading conventions.

The PCOs reported less frequently included QOL, physical discomfort, patient values and experiences, patient financial and time costs, psychosocial outcomes, and test durations. Patient-reported questionnaires were the most common tool used to capture outcomes of QOL, emotions (eg, anxiety), patient values and experiences (eg, discomfort during imaging, patient satisfaction, patient preference), and physical discomfort and psychosocial outcomes (eg, pain, claustrophobia, depression) at defined time points after imaging. Validated questionnaires were used to solicit QOL PCOs. Patient and financial time costs were measured using a mixture of patient report, administrative claims records, and calculating the time the patient spent on diagnostic workup relative to their net income.

Discussion

The primary studies used to inform the AC recommendations appear to be narrowly focused and contain relatively few PCOs. The outcomes identified from the 89 clinical topics reviewed were mainly related to immediate or short-term adverse physical effects from the test process itself, such as adverse reactions or radiation exposure. We found a small amount of evidence for other outcomes, including outcomes related to incidental findings or emotional or behavioral effects. Among all outcomes identified, only a very small number were patient reported. Given that measurement of, reporting of, and inclusion of PCOs in these clinical guidelines are uncommon, our findings highlight the importance of the PROD study and its recommendations regarding the proposed incorporation of PCOs into study design, reporting, and guidance for clinicians and patients.

A strength of the PROD study was the use of the AC evidence tables, as they are widely used in radiology and span a large number of imaging modalities and clinical areas. Other sources of synthesized evidence might have provided different primary studies with potentially different findings, and the ACR evidence may not be representative of other sources of evidence on imaging tests in the United States. We used systematic methods to screen for eligible articles in accordance with standard evidence synthesis methods. The main limitation of this approach is that the AC evidence tables focus on test accuracy and were not created with a wider range of outcomes in mind. We found very few study designs outside diagnostic accuracy studies, which suggests that the types of studies where PCOs are likely to be measured—qualitative studies or surveys—had been excluded in the ACR process. We were surprised that neither the known evidence on the psychological harms of screening studies (at least for mammography) nor the known studies on the reassurance provided by imaging tests were identified in the AC evidence.^147-149

Other limitations included the highly heterogeneous studies reviewed, which precluded a meta-analysis; the fact that the systematic reviews used to generate the AC evidence tables were conducted at different times; that they did not list all the search terms used; that searches were limited to the MEDLINE database; and that the AC do not list reasons for exclusion of studies.

Summary

This secondary analysis of a commonly used set of clinical guidelines for informing the use of imaging studies across a very wide range of clinical indications and imaging modalities found that few PCOs are reported in the literature that is used to inform these recommendations. The PCOs that are mentioned largely focus on adverse reactions and radiation exposure. This contrasts with the wide range of PCOs identified in the scoping review and suggests that processes for searching for and/or including this literature within the ACR AC are insufficient. The analysis also emphasizes the need for the methods work that the PROD study recommendation aims to inform, so that PCOs can be included in guidelines used by clinicians and patients to make decisions about imaging test selection.

Background

Note: Some of the text in this “Background” section is adapted from Thompson MJ, et al. Patient-centered outcomes of imaging tests: recommendations for patients, clinicians, and researchers. BMJ Qual Safety (under review). Copyright permission will be requested once the manuscript has been published. [PMC free article: PMC10447372] [PubMed: 34615733]

Over a 3-year period, the PROD study conducted primary research and evidence synthesis and gathered input from multiple stakeholders to develop new, consensus-based recommendations for the use of PCOs in imaging test evaluations. We anticipate that this expansion of methods regarding diagnostic test evaluation will stimulate new standards for research, reporting, and use of PCOs in imaging tests, and potentially in diagnostic tests, more broadly.

As summarized in this section (see Figure 4), our approach was based on the multistep processes used to develop consensus methods and research reporting guidelines.^{15,23,150,151} Over the study period, the PROD team (1) defined the methodology gap and identified relevant literature on existing models and frameworks, (2) recruited and engaged multiple stakeholders, (3) determined the current frequency of use of PCOs in recommendations for imaging tests in clinical practice, (4) conducted primary qualitative research with patients and health care providers to identify PCOs, (5) expanded the qualitative findings by exploring the wider literature in a scoping review, and (6) developed a consensus-based recommendation on the use of PCOs in evaluations of imaging tests¹⁵² (see Figure 4). Steps 1 through 5 are covered in previous sections of this report. A scoping review of qualitative studies that had reported PCOs from imaging studies was used to provide a broader evidence base for the range of PCOs identified from qualitative studies. This section explores how we implemented step 6, developing consensus-based recommendations.

Figure 4

Data Sources and Methods Used to Develop Final Recommendations.

We considered several different approaches to developing this methods recommendation and the scope of the planned recommendation. The approach we used was based on similar processes used to develop research reporting guidelines and NIH consensus statements. We opted for a broader, more comprehensive approach, however, informed not only by evidence from primary and secondary research but also by extensive input from stakeholders. Within the overall approach to imaging tests, we attempted to balance breadth (ie, not focusing on 1 type of imaging test or clinical scenario) and depth (ie, attempting to define in sufficient detail outcomes that could be broadly applied). We opted not to use Delphi or nominal group methods to rank or achieve consensus.¹⁵³

Consensus Statement Drafting and Refinement

The stakeholders, whose guidance shaped our research and examined our results, were critical to developing statements and recommendations drawn from our findings. At every step of the PROD study, they provided valuable feedback. They reviewed and provided feedback on PCOs that we identified as already in use through our literature reviews, secondary analysis of ACR's Diagnostic Imaging AC, and a scoping review of qualitative studies. The stakeholder advisory board also provided feedback on our semistructured interviews to identify PCOs and contributed to the interpretation of this data.

Toward the end of the 3-year period, stakeholders participated in a 2-day in-person meeting held in Seattle, Washington. The meeting used NIH Consensus Conference methods to provide input and feedback on the statement of proposed PCO domains and recommendations.¹⁵² A total of 28 individuals participated in this meeting (Appendix). In the 2 months before the meeting, stakeholders received preparatory material in the form of published papers (or drafts) of research conducted by the PROD study group and summaries of other key publications. The interactive meeting included presentations of key findings, smaller-group breakout sessions, and large-group discussions, with the goal of answering the following 4 sets of questions: (1) Which PCOs are important to patients? How and when should PCOs be measured in studies of imaging tests? (2) What should researchers do differently? What is transferable across all types of diagnostic tests? (3) How can PCOs be implemented in tools and strategies to inform doctor-patient communication and decision-making? and (4) What methods and strategies should be used to disseminate our findings and future recommendations? Each question was posed for a breakout session and was discussed in smaller groups. The groups came together to share key points from their discussions with the broader group. The large group compared overlap of key points from the small breakout discussions and further discussed points that diverged until consensus was reached. Following the meeting, stakeholders reviewed drafts of the consensus recommendations, and agreement was reached on the final document.

Results

Note: Some of the text in this “Results” section is adapted from Thompson MJ, et al. Patient-centered outcomes of imaging tests: recommendations for patients, clinicians, and researchers. BMJ Qual Safety (under review). Copyright permission will be requested once the manuscript has been published. [PMC free article: PMC10447372] [PubMed: 34615733]

Main domains of PCOs

PCO domain 1: Information or knowledge gained from an imaging test. Outcomes within this domain includes test information that leads to finding the cause or underlying diagnosis for patients' symptoms or concerns, as well as information that leads to lowering the possibility of a condition that the patient or provider was concerned about (Table 3). In addition to these, patients described the “value of knowing” (and for imaging tests, the “value of seeing”), regardless, to some extent, of the outcome of the test, and appreciated the additional information that the test revealed. Closely related to these were outcomes pertaining to the impact of the information yielded by the test on decision-making, such as being able to access a higher level of care or to proceed with a particular treatment course. Negative outcomes within this domain included misleading information from the test, particularly, false-positive test results that led to the need for additional testing (or a sequence of tests) to further confirm/rule out a condition as well as false-negative test results and inconclusive or indeterminate results that did not provide a definitive diagnosis and also often led to further testing. Unexpected findings that might have clinical significance but could lead to additional testing or investigations, including detection of incidental findings (ie, results that might require action but were not expected from the imaging), were also reported.

Table 3

Outcomes Within the Domain of Information or Knowledge Gained From Imaging Tests.

PCO domain 2: Physical effects of the test or testing process. Preparation for some imaging tests led to physical outcomes, such as unpleasant experiences from undergoing bowel preparation needed before testing (Table 4). However, outcomes that are more prominent were pain and physical discomfort during the imaging procedure itself as well as its immediate consequences. In some instances, discomfort caused by the same imaging modality depended on the area of the body being examined. Other outcomes within this domain included the effects of ionizing radiation and potential adverse effects of contrast material. Exposure to ionizing radiation, which was typically cited only by patients undergoing frequent imaging during cancer therapy, was not a prominent concern reported by patients, yet it is one that is commonly reported in the literature. This may indicate that patients have challenges perceiving the longer-term potential impacts of radiation, compared with shorter-term, more immediately noticeable effects. Potential adverse effects of contrast material were noted by health care professionals and measured in several studies, but they were only occasionally noted by patients themselves.

Table 4

Outcomes Within the Domain of Physical Outcomes From Imaging Tests.

PCO domain 3: Emotional impact of the test or testing process. Imaging tests can lead to both positive and negative emotional outcomes (Table 5). When testing was used to evaluate new and/or concerning symptoms, testing led to outcomes of reassurance and relief when results ruled out certain conditions, and when results indicated an underlying condition, testing led to relief that the cause of the patients' concerns or symptoms had been found. Testing leads to outcomes, such as anxiety, however, not only before the test procedure (anticipation anxiety), but more prominently while waiting for test results or learning of serious (including initially false-positive) test results. During the imaging procedure, the physical constraints, narrowness of the imaging scanner, noise, and unfamiliarity of the procedures can lead to feelings of claustrophobia and distress or embarrassment from having to undress in front of health care staff. On some occasions, negative emotional outcomes may occur from disappointment or regret in entering a testing pathway in the first place, such as disappointment with negative test results that do not answer patients' concerns or that fail to reveal information that advances their diagnosis or clinical care.

Table 5

Outcomes Within the Domain of Emotional Outcomes From Imaging Tests.

PCO domain 4: Test burden. A final group of outcomes is related to the direct or indirect costs and burdens related to imaging tests (Table 6). Undergoing imaging testing may incur direct costs to the patient in terms of out-of-pocket expenses of the imaging study or with insurance co-pays or insurance deductibles, but this varies with the health care/health insurance system. These costs were noted when complex imaging tests (MRI, CT scan, etc) were used or when a series of less expensive imaging studies revealed findings that subsequently led to more complex and more costly types of imaging testing. Direct out-of-pocket costs may not be expected or considered before the test is performed. Indeed, not all patients recognized or were aware of the costs of imaging. Similarly, health care providers recognized the importance of costs to patients but often could not identify these costs given the complexity of the health insurance system. Other test burdens included time, such as waiting times after a test had been ordered, and the time incurred in attending for the actual imaging procedure, such as time away from work, which was noted particularly for patients living long distances from imaging centers (eg, in rural areas). In addition, the burden of time occurred in the waiting period for test results. Very occasionally, the results of imaging testing could lead to direct financial benefits to patients, for example, when imaging testing revealed conditions related to occupational health conditions, such as occupational lung disease or workplace injuries, or when it was linked to applications for disability benefits.

Table 6

Outcomes Within the Domain of Burden From Imaging Tests.

Interactions between outcomes

In addition to the outcomes grouped within the above-mentioned 4 domains, it is clear that PCOs interact with and potentially influence outcomes in other domains. For example, knowledge or information gained from a test result may influence patient emotions, such as providing reassurance or provoking feelings of anxiety. Also, a test that is more physically unpleasant might provide more valuable information than does one that is less invasive. The pattern of interactions is complex and likely varies with test modality, clinical situation, etc. Recognizing that interactions occur suggests that weighting of risks and benefits across domains is likely to be challenging to incorporate into research or clinical care.

Modifiers of outcomes

Multiple contextual elements potentially affect or modify whether a PCO occurs and its severity, relative importance or weight, and impact on the patient (Table 7). These range from the characteristics of the individual patient to the type of test they are undergoing, the clinical situation, the health care team involved, the physical environment of the testing suite, and communication of the results. Embedded within several of these modifiers is the concept of a patient's prior (or pretest) probability of a particular outcome or condition. For example, a patient with a known history of a given condition may be at higher risk of recurrence (higher prior probability), whereas an individual being screened (without symptoms of that condition) would be at a lower prior probability. These prior probabilities could influence outcomes following the test result (such as greater anxiety in the emotion domain). In addition, the prior probability of a given condition may vary with health care setting and individual clinician knowledge (or perception) of a given outcome or condition, leading to potentially further modifying PCOs.

Table 7

Factors That May Modify PCOs Related to Imaging Tests.

Discussion

The PROD study consensus provided the first comprehensive attempt to develop a comprehensive definition of PCOs as they pertain to imaging testing and make recommendations for steps needed to implement them within clinical practice for patients considering and/or undergoing imaging tests. We used an approach that was broader and more comprehensive than the one typically used to develop research reporting guidelines. We included evidence from several sources, namely, (1) published literature in this area, (2) primary qualitative research from patients and providers on PCOs, (3) a scoping review of the wider qualitative literature on PCOs related to imaging tests, (4) a secondary analysis of the current use of PCOs in radiology research recommendations, and (5) extensive input from stakeholder groups.

The focus of this study is imaging testing, which is a broad area in itself, with a conscious choice to balance breadth (ie, not focusing on 1 type of imaging test or clinical scenario) and depth (ie, attempting to define in sufficient detail the outcomes that could be broadly applied).

There are some limitations to the findings from this consensus process. First, the clarity of definitions of outcomes within each domain varied to some extent. We appreciate that some PCOs may require tighter definitions or may need to be modified as this area of research on imaging tests (and potentially other types of diagnostic tests) expands. Second is that our focus, as noted earlier, was on the more direct outcomes of tests, including immediate consequences of a test (such as additional testing needed after a false-positive test result). We acknowledge that tests form part of a patient's journey and that separating the outcomes related to diagnostic testing is to some extent artificial. The contrary argument is that focusing (as is done currently) on the downstream/clinical end points often ignores the direct effects of the test. We also feel that numerous other methods experts and reports, as described in the “Background” section, have identified the need for the research and recommendations we have proposed here.

The 3 main statements regarding the definition of PCOs that emerged from the consensus meeting were that PCOs from imaging studies fall within 4 main domains: (1) information or knowledge gained from the imaging test, (2) physical effects, (3) emotional outcomes, and (4) test burden. We acknowledge that not all outcomes within each of these domains can be defined with the same precision. Moreover, outcomes interact with and/or influence outcomes in different domains in ways that are likely to be complex and to vary with test modality, clinical situation, etc. Finally, we noted that PCOs are not experienced identically but rather are modified or mediated by factors related to the patient, the clinical environment, and the physical environment of the test.

The main recommendations from the consensus meeting that addressed implementation of the use of PCOs to inform clinical decision-making were as follows:

1. Research is needed to determine the relative importance of various outcomes to each other (and to test accuracy) so as to balance beneficial and harmful outcomes, acknowledging that these are likely to vary not only with the severity/prominence/impact of the PCO related to that test, but also with the various factors that modify or mediate this outcome.
2. Validated measurement instruments or tools need to be created (or modified from existing tools) to measure PCOs, including those that collect information from patients themselves.
3. Research is needed to determine the extent to which PCOs are transferable across different imaging modalities and different clinical applications of a given imaging modality. While the current report focused on imaging testing, further research will be needed to determine the generalizability of the PCO domains to other types of diagnostic testing.
4. A reporting guideline that incorporates PCOs as outcome measures within a diagnostic test evaluation may be valuable to improve reporting of studies detailing additional test outcomes.
5. Patients and their health care providers need information on PCOs relevant to their testing situation. This information should be collected, collated, and presented in a way that can inform decision-making without adding burden to patients and providers.

The PROD study provides the first comprehensive attempt to define PCOs of imaging tests and to propose next steps needed to incorporate these outcomes into research and practice in imaging testing, and potentially the broader field of diagnostic testing.¹⁷³ Our findings add to models that first proposed PCOs related to diagnostic tests.⁷ The harmful impacts of screening tests on emotional outcomes, particularly from false-positive test results, have been described extensively,^147,174 yet beyond these specific test uses, they are largely overlooked.¹⁷⁵ Moreover, focusing exclusively on the harms of tests ignores their potential benefits³⁵ and thus prevents a balanced assessment. The proposed domains largely map onto the effects of tests proposed by Bossuyt and McCaffery⁷ (which consisted of emotional, social, cognitive, and behavioral effects), with the exception of the impact of tests on behavioral outcomes, for which we found little evidence in the imaging literature. The domains of patient-reported outcomes within the NIH Patient-Reported Outcomes Measurement Information System (PROMIS®) have focused on outcomes related to treatment interventions and include measures within the 3 domains of physical health, mental health, and social health.¹⁷⁶

We believe there are several reasons why examining PCOs should now be considered an essential part of test evaluations. Numeric measures of test accuracy, taken in isolation from other effects or outcomes, risk ignoring outcomes of a test that may have significant meaning from the perspective of the individual patient. Rather, the balance of outcomes (or test effects) is important to patients (and their clinicians). For example, a highly accurate but invasive test that poses significant risks of adverse effects (eg, pain, bleeding, etc) needs to be balanced against a test with inferior accuracy but fewer adverse effects or greater availability.⁹ So little is understood about the benefits and harms of diagnostic tests,^30,155 however, that patients and their clinicians simply do not have enough information to make rational choices in selecting or avoiding tests.

The use of PCOs in diagnostic testing has implications for guideline developers and health service policy makers. Measures of patient satisfaction are now common in most health care systems, yet they typically only focus on patient expectations of the services and experience provided, overlooking patient outcomes. Some evidence exists, however, that fulfilling outcomes that are important to patients may lead to greater satisfaction. This can only be achieved for diagnostic testing if the outcomes that are important to patients are known and can be measured.^170,177, Guideline developers have been more aware of the potentially nuanced balances of test outcomes, particularly in the area of screening tests. In their attempts to weigh the risks and benefits of tests, however, guideline development groups have been hampered by a lack of data on PCOs from primary research studies. Finally, regulatory approval of new tests typically has focused on test accuracy, but there is growing interest by the FDA in considering additional impacts of tests on approval decisions.