An official website of the United States government
NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.
Deborah R Maiese, MPA, Tabitha P Hendershot, BA, Lisa C Strader, MPH, Diane K Wagener, PhD, Jane A Hammond, PhD, Wayne Huggins, PhD, Richard K Kwok, PhD, Dana B Hancock, PhD, Nedra S Whitehead, PhD, Destiney S Nettles, BA, Joseph G Pratt, MPM, Marcia S Scott, PhD, Kevin P Conway, PhD, Heather A Junkins, MS, Erin M Ramos, PhD, and Carol M Hamilton, PhD.
The PhenX (consensus measures for Phenotypes and eXposures) Toolkit offers well-established, broadly validated measures of phenotypes and exposures relevant to investigators in human genomics, epidemiology, and biomedical research. This methods report describes the infrastructure and processes used to develop the content and features of the Toolkit. The PhenX consensus process is robust, yet flexible, as evidenced by its application to a range of research domains.
During the initial phase of PhenX, from March 2008 through April 2010, working groups of content experts addressed 21 research domains and selected 295 measures for the Toolkit. The PhenX Steering Committee prioritized and defined the scope of the domains and guided the consensus process with input from liaisons representing the National Institutes of Health. After the 21 domains were completed, another project to add breadth and depth to the Toolkit for substance abuse and addiction (SAA) research served to validate the consensus process. With the support of the SAA Scientific Panel to define the scope for one core and six specialty collections and SAA working groups to select measures, the PhenX project team added 44 measures to the Toolkit in 2012.
Now being used by more than 1,000 researchers, the PhenX Toolkit offers a catalog of measures, supporting documentation, and tools for collaborative research. It used a consensus process that can serve as a template for investigators who are considering a similar approach.
Funding for this work was provided by the National Human Genome Research Institute (NHGRI) of the National Institutes of Health (NIH) Cooperative Agreement U01 HG004597 with supplements from the Office of Behavioral and Social Sciences Research and the National Institute on Drug Abuse (NIDA). The authors thank the members of the PhenX Steering Committee, the Substance Abuse and Addiction Scientific Panel, the NIH liaisons, and the working group members for their many contributions. The authors also thank Wally Campbell, Jennifer Drolet, and August Gering for editorial assistance and RTI’s Creative Services group for graphics assistance. The views and opinions expressed in this report are those of the authors and should not be construed to represent the views of NIDA, NHGRI, the National Institute of Environmental Health Sciences, or any of the sponsoring organizations, agencies, or the US government.
The National Human Genome Research Institute (NHGRI) of the National Institutes of Health (NIH) recognized the need to identify and promote well-established measurement protocols that are useful in genome-wide association studies (GWAS) and other human subjects research. The recommendation for a “preferred set of data elements” originated at the 2006 Multi-IC [NIH Institute and Center] Symposium on the Application of Genomic Technologies to Population-Based Studies. The following year, the Frontiers in Population Genomics workshop reiterated the recommendation to identify a subgroup of phenotypes and exposures for standardization and addition to GWAS (National Human Genome Research Institute, 2007). To implement these recommendations, NHGRI released a request for application (RFA) for a web-based resource of high-priority measures that researchers could use for cross-study comparisons, integrated data analysis, and validation studies (NIH, 2007).
In response to this RFA, RTI International proposed a consensus process to engage content experts in the development of a web-based resource of high-priority measures. A steering committee would provide guidance, and working groups of content experts would select the measures. Liaisons from the NIH ICs and the broader scientific community would be engaged and asked to provide input during the measure-selection process. All decisions by the steering committee and working groups would be established by consensus. This approach was chosen to ensure that the measures and the tools used to present the measurement protocols would address the needs of the scientific community.
As set forth in the NHGRI RFA, the objectives of the research program were as follows:
This paper describes the infrastructure, selection criteria, prioritization processes, and outreach methods used to engage the scientific community in identifying high-priority measurement protocols for inclusion in the PhenX (consensus measures for Phenotypes and eXposures) Toolkit (www.phenxtoolkit.org ). A discussion of the advantages and challenges of the approach demonstrates its flexibility and applicability in various situations. This methodology may be suitable for a variety of applications where experts in the field seek broad approval from the scientific community. Refer to the appendix for a glossary of PhenX terms and acronyms (also available on the PhenX website under the Resouces tab).
The goal of PhenX is to provide investigators with high-quality, low-burden measures for inclusion in GWAS and other human subjects research (Hamilton et al., 2011). This consensus process was applied to 21 research domains as well as to seven substance abuse and addiction (SAA) collections of measures.
Building consensus for PhenX required substantial involvement of the scientific community. The project enlisted investigators to identify well-established protocols and to promote the use of common measures for collaborative research. The project infrastructure included a steering committee to provide overarching policy guidance and expert, domain-specific working groups to identify common measures and protocols for widespread recommendation. NHGRI scientists had already engaged their colleagues at NIH in laying the foundation for the project (National Human Genome Research Institute, 2007) and providing input on the RFA.
Often, the federal government uses consensus processes to build support for important scientific policy. For example, many NIH ICs convene expert panels to address scientific issues on the diagnosis and treatment of disease. The NIH also has a Consensus Development Program that convenes experts, according to its website, “to evaluate the available scientific information and develop a statement that advances understanding of the issue in question and will be useful to health professionals and the public at large” (http://consensus.nih.gov).
The National Academy of Sciences and the Institute of Medicine, at the request of federal agencies, invite experts to serve on study committees to reach consensus in independent and objective reports (Division on Earth and Life Studies of the National Academies, 2013).
RTI developed the PhenX consensus process to address a specific goal: to select common measures for use in human subjects research. As shown in the organizational chart for the PhenX project (Figure 1 ), NHGRI was responsible for guiding the public investment in the project. The RTI project team, led by a principal investigator and co-investigator, followed the research plan laid out in its proposal and was responsible for directing the day-to-day operations of the PhenX project. The project team, with significant involvement from NHGRI, recruited the steering committee to guide the consensus development process and engaged NIH liaisons and the scientific community in the development of the PhenX Toolkit.
PhenX organizational chart. NHGRI = National Human Genome Research Institute; IC = National Institutes of Health Institutes and Centers; WG = working group; SSP = Substance Abuse and Addiction Scientific Panel.
The PhenX experience provides a template for other collaborative efforts that would benefit from a consensus approach.
The NHGRI used a cooperative agreement (U series) support mechanism for the PhenX project that, by definition, indicates “substantial Federal scientific or programmatic involvement . . . [to] assist, guide, coordinate, or participate in project activities” (NIH, 2013). Cooperative Agreement U01 HG004597 was launched to establish common measures for 20 high-priority research domains. The NIH Office of Behavioral and Social Sciences Research provided support for a 21st domain, Social Environments.
At the outset, the NHGRI project scientist notified NIH staff that an award had been made to RTI. As a follow-up to that e-mail notification, the PhenX investigators scheduled telephone calls to seek participation from the NIH ICs. The investigators asked the ICs to help ensure that the Toolkit would meet the needs of their scientific constituencies and be coordinated with other NIH measurement projects. IC officials were asked to
The PhenX project team developed a fact sheet about the roles and responsibilities of NIH liaisons. Some NIH liaisons had to obtain formal approval to participate. The 22 NIH liaisons are identified on the PhenX NIH liaisons roster on the PhenX website.
Throughout the project, the NIH liaisons were invited to participate as nonvoting members in steering committee and working group meetings. Some NIH liaisons also served as bona fide (voting) working group members, sharing their substantial expertise and experience and helping ensure that the measurement protocols were well established and useful for the foreseeable future in the Toolkit.
The liaisons helped coordinate PhenX with other NIH initiatives, such as the NIH Toolbox and Patient-Reported Outcomes Measurement Information System (PROMIS; Riley, Pilkonis, & Cella, 2011 ), and with bioinformatics efforts, such as the database of Genotypes and Phenotypes (dbGaP; Mailman et al., 2007), the Logical Observation Identifiers Names and Codes (LOINC; Vreeman, McDonald, & Huff, 2010), and the Cancer Data Standards Registry and Repository (caDSR) of the cancer Biomedical Informatics Grid (caBIG; caBIG Strategic Planning Workspace, 2007).
At the outset of the project, PhenX project staff established a 12-member steering committee to
PhenX staff recruited the steering committee members to reflect a broad spectrum of research and the diversity of domains envisioned for the Toolkit. During telephone conversations, the RTI consensus coordinator posed questions to potential steering committee members about the proposed criteria for prioritizing domains, asked for their recommendations for the first five domains, and requested suggestions for what end products would be useful to meet the needs of the research community. These discussions helped to highlight issues to be addressed in the early phases of the Toolkit’s development.
As Table 1 shows, the steering committee chair and eight other steering committee members were recruited from academia; one retired NIH employee served as the steering committee vice chair. The NIH project scientist was a voting member of the steering committee, as was a member of the RTI staff. The first meeting of the steering committee took place in January 2008. The steering committee held three face-to-face meetings per year in Washington, DC, and conducted conference calls between these meetings. Over a 4-year period, the steering committee was convened 18 times.
Phenotypes and eXposures (PhenX) project steering committee members.
The roles of the steering committee were to define the content scope of each research domain, to identify expert working group members to address and refine the broad content initially set forth in the scope document, and to serve as liaisons to the working groups. The steering committee policy guidance was reflected in the PhenX Guidance Document for Working Group Members, which provided a standardized approach to working group deliberations (e.g., who is in the working group, how the working group will work, and timeframes for accomplishing the tasks). The Guidance Document included definitions (e.g., “what is a measure”) and materials to guide the working group process (e.g., criteria for evaluating measures). As the project progressed, the steering committee was called on to address new policy issues, which resulted in updates to the Guidance Document.
To provide guidance to the working groups during their discussions and deliberations, one member of the steering committee volunteered to be the steering committee liaison for each working group. The steering committee liaison was a nonvoting member of the working group who provided the working group with perspective and policy advice. The steering committee liaison brought matters that could not be resolved by the working group to the steering committee for discussion and resolution.
The steering committee reviewed all measures before community outreach and before the measures’ release in the PhenX Toolkit. This ensured that the process was consistent across working groups and that each working group chose measures that sufficiently addressed the scope of the research domain. Steering committee members also contributed to the overall design of the PhenX Toolkit and suggested features to make the web-based resource easy for investigators to use.
The NHGRI project scientist served on the steering committee, having input into the policies and plans that guided the project. She also served as the steering committee liaison to the Anthropometrics working group, one of the first three working groups to be convened. Biweekly meetings occurred between the NHGRI project scientist and the RTI PhenX leadership to address issues as they arose from the steering committee and working group deliberations. The NHGRI project scientist initiated communications with NIH colleagues to ensure coordination between this effort and other measurement initiatives. The NHGRI project scientist also sent the e-mails to potential steering committee and working group members to request their service in identifying measures and protocols for the Toolkit.
Project staff established PhenX working groups, with each comprising six to nine scientists with relevant domain expertise, for each of the 21 domains. The roles of each working group were to
The steering committee domain scope guided the recruitment of experts. Steering committee members and NIH liaisons identified scientists to recruit; RTI staff identified others through literature searches. The steering committee shared lists of potential participants with the project scientist, who sent an initial e-mail to introduce the PhenX project. Often project staff recruited working group chairs first and gave them the opportunity to suggest potential working group members. Five working groups had co-chairs, an organizational approach that helped with logistics and provided complementary leadership and expertise. Complete lists of working group experts by domain are on the PhenX website.
PhenX staff identified individuals for their expertise, not their organizational affiliations, although they usually selected only one person from a given institution. As per guidance from the steering committee, each working group included at least one scientist with significant genetics/genomics experience. For 18 of the 21 domains, NIH experts from the relevant ICs served as working group members. As the working groups were assembled, PhenX staff made a concerted effort to include diverse disciplines; maintain a balance of clinician scientists (i.e., MDs) and academic researchers (i.e., PhDs); include both senior and junior investigators; and encourage gender, racial, and ethnic diversity. The formation of the working groups was an interdisciplinary effort that involved investigators from across the United States.
The roster of potential participants included double the number of scientists who would ultimately fill the working group. This approach took into account coverage of the scope and potential attrition due to scheduling conflicts. The RTI consensus coordinator had a single telephone conversation with each of the potential working group members. Each discussion focused on the project and the researcher’s perspectives on prioritizing and achieving consensus on measurement protocols for that domain. An understanding of data harmonization challenges and a shared vision about the importance and benefits of common measurement increased the likelihood of being recruited for the interdisciplinary working groups.
People who supported the goal of a common currency for interdisciplinary research and who understood the challenges of data harmonization readily accepted the invitation to participate in the PhenX project. Because the purpose of PhenX was to select from among existing measurement protocols, a few researchers who indicated a preference to develop new measures did not participate.
Developing an effective consensus process required considerable RTI staff involvement, as shown in the RTI PhenX organizational chart (Figure 2). The principal investigator, co-investigator, and project manager made up the core management team. The principal investigator was responsible for the Toolkit development team, communications team, and logistics team and for oversight of the entire project. The co-investigator was responsible for the consensus coordinator, the steering committee coordinator, the working group supervisors, and the working group managers. The project manager coordinated the administrative functions of the project, including the budget, timelines, and logistics. The project leadership met weekly to discuss internal management issues. They also met biweekly with the project scientist and other NHGRI staff. To ensure effective communication across the project, the co-investigator held weekly meetings with the consensus coordinator, working group supervisors, and working group managers. These meetings facilitated discussions of policy and scheduling issues as they arose and were essential to implementing consistent methodology across the working groups. Many of the working groups were running concurrently, which added logistical challenges as the result of managing multiple domains at various stages of the process.
Organizational chart for the RTI PhenX team.
Coordination between the steering committee and working groups was critical. RTI staff conducted pre-briefings with the steering committee chair and vice chair before every steering committee meeting. Similarly, calls with the working group chairs and the designated steering committee liaisons occurred before every working group meeting to address emerging issues and to facilitate successful and efficient meetings. This intensive staffing effort was designed to minimize the workload on the steering committee and working group members and to provide them with agendas and materials to accomplish the tasks at hand. Over time, RTI staff established a working group process that included standard operating procedures for tasks and templates for documentation. RTI provided minutes for all steering committee and working group meetings.
Aside from the management and working group support teams, three other teams were needed. The RTI communications team was responsible for maintaining Listservs, conducting outreach with the scientific community, producing the PhenX newsletter, and maintaining the PhenX project website and a private portal as a workspace for the steering committee and working groups. The logistics team was responsible for the meeting venues (e.g., hotel, conference rooms, telephone, and audio/visual equipment) for the 8 in-person steering committee meetings, 1 in-person SAA Scientific Panel (SSP) meeting, and 22 in-person working group meetings (two working groups did not hold an in-person meeting due to logistic and weather restrictions). The Toolkit development team was responsible for the design, development, and maintenance of the PhenX websites, www.phenx.org and www.phenxtoolkit.org, which provide information to the public and private work areas for the steering committee and working groups.
At the first in-person steering committee meeting in January 2008, the steering committee discussed whether the domain list should be disease oriented, risk-factor oriented, or based on a model from the World Health Organization of physical, mental, social, behavioral, and environmental domains. As the steering committee nominated domains, some were organ-based domains (e.g., Gastrointestinal and Ocular), and others were disease-based domains (e.g., Cancer; Diabetes); some represented specific exposures (e.g., Alcohol, Tobacco and Other Substances; Environmental Exposures); and some examined health determinants (e.g., Physical Activity and Physical Fitness; Nutrition and Dietary Supplements). The steering committee used the following criteria for selecting the domains:
This framework guided the organization of the PhenX working groups (Table 2 ). The steering committee chose the following as the first three domains: Demographics, Anthropometrics, and Alcohol, Tobacco and Other Substances. These domains were seen as foundational to virtually all research and were selected as the building blocks for the Toolkit. Cardiovascular was the first working group convened that was related to an organ system. As the project progressed, the steering committee recommended convening working groups for domains that should be addressed simultaneously to avoid potential overlap of selected measures and encourage collaboration, such as the domains Social Environments and Psychosocial. The 21 domains selected for the Toolkit cover a broad scope of biomedical research.
PhenX Toolkit domains, number of measures, and Toolkit release date.
The steering committee organized the scope of the domains into the following categories:
The intent was to help the steering committee consistently define the scope of each domain and identify areas of expertise that needed to be recruited for each working group.
PhenX staff convened working groups for 21 domains (see Table 2) between March 2008 and April 2010. The steering committee tasked the working groups with selecting up to 15 high-priority measures, recommending protocols for the measures, and ensuring inclusion of the information needed to reliably replicate collection of the data.
Each working group reviewed and, to some extent, modified the original scope of the domain provided by the steering committee. These scope modifications were subject to review and approval by the steering committee. Several working groups also refined the names of their domains to better reflect the scope of the measures proposed for the Toolkit. Typically, each working group broadly assessed measures relevant to its domain before beginning the prioritization task. Each working group was to select up to 15 measures for inclusion in the Toolkit, which was a significant challenge based on the broad scope of these research domains.
The working group process began with an introductory conference call, followed by an in-person meeting and a minimum of three teleconferences. Additional teleconferences, e-mails, and portal discussions were conducted as needed. From the first call until the final teleconference, the working group, under the leadership of the working group chair/co-chairs, sought to achieve consensus regarding the measures that would cover the domain’s scope (see Figure 3) (Hamilton et al., 2011).
Standard PhenX consensus process.
Attending the working group in-person meeting was critical because working group members made key decisions at this meeting about which measures met the criteria for inclusion in the Toolkit. The all-day, face-to-face meeting facilitated participation, discussion, and resolution of challenging issues and moved the working group toward consensus. Each working group was assigned an RTI working group supervisor and working group manager who assisted the working group with the preparation of agendas, meeting summaries, measurement protocols under consideration, and other materials to provide efficient support for the working group’s deliberations. The working group supervisors served as the scientific liaisons to the working groups and helped the working group managers anticipate problems, proactively develop strategies to resolve issues, and keep to the schedule in order to guide the working groups to consensus in their choices of measures and protocols.
Workgroup members identified measurement protocols through various means. Staff at the NIH National Library of Medicine conducted searches in dbGaP (the NIH repository for GWAS), and RTI staff reviewed the literature for additional protocols. Working group members were also responsible for identifying protocols and making presentations about their recommendations on the measurement approach at the in-person meeting. Selection criteria (see box) for the measurement protocols included validity, reproducibility, feasibility, and low burden to both investigators and study participants. Some working groups quickly applied these criteria and arrived at a set of measures that covered the scope of the domain. For other working groups, the process proved more challenging and required significant facilitation by the steering committee liaisons, working group supervisors, and working group managers.
Criteria Used for Selecting PhenX Measures.
The scientific community participated in an outreach effort to review and comment on the proposed measures and protocols for every domain. The PhenX communications team gathered input from the working groups, steering committee members, and IC liaisons to create a Listserv for the outreach efforts. To raise awareness and get additional input, the outreach also involved requests to research organizations and their Listservs and to the professional networks of the working group members and PhenX staff.
Over a span of 10 business days, the scientific community was able to comment on each measure and its associated protocol. For those interested in specific details, RTI provided a link to comprehensive draft datasheets for each of the proposed measures, which presented the information about the measures as it would appear in the Toolkit. The datasheets were developed by RTI staff, carefully reviewed by working group members, and approved by the steering committee prior to release in the Toolkit.
RTI staff compiled summaries of outreach feedback that facilitated the working groups’ final deliberations. The working group chairs made a final presentation to the steering committee on the set of measures and protocols. The steering committee approved the set before it was publicly released in the Toolkit. Between 2008 and 2012, more than 1,000 scientists (including the steering committee, expert working groups, and respondents to outreach) participated in the selection of the measurement protocols for the PhenX Toolkit.
In 2011, the NIH National Institute on Drug Abuse (NIDA) sought to expand the breadth and depth of the 26 measures in the Toolkit that addressed alcohol, tobacco, and other substances. Although the scope and purpose of the SAA project were distinct from the initial Toolkit development, the PhenX project team implemented the same consensus process with some slight modifications and with an expedited timeline.
Unlike the initial Toolkit development, which aimed to provide a few measures relating to 21 broad research domains, the SAA project sought to add measurement protocols to serve a specific research constituency—SAA researchers. To ensure that the SAA content would be consistent with the overall goals of the PhenX Toolkit, the SAA team included the steering committee in the development of the SAA research plan. The more focused scope of the SAA project required that the PhenX team recruit a 10-member SSP (see https://phenx.org/Default.aspx?tabid=689) to provide guidance while the steering committee continued to provide oversight.
The SSP defined the scope of the six specialty collections and tasked three working groups to each address two specialty areas. The six specialty collections and the assigned SAA working groups are shown in Table 3.
Substance abuse and addiction working groups and six specialty collections.
Before identifying a core collection of SAA measures, the SSP reviewed the existing PhenX measures and the newly proposed SAA measures. NIDA recommends that all substance abuse researchers conducting human subjects studies use the core collection of measures (NIDA, 2012). The SAA project added a total of 44 new measures to the Toolkit on February 24, 2012.
The SSP preserved many key components of the PhenX consensus process and successfully replicated them in selecting the SAA measures. These include the following:
The organizational structures (see Figures 1 and 2) were essential to collaborative decision making and ongoing communications between the NHGRI project scientist, the RTI team, and the steering committee. The steering committee provided overall guidance to the working groups, including the criteria for choosing the measures (see Table 4). All 21 domain working groups and the SAA project used the PhenX Guidance Document for Working Group Members. The Guidance Document template worked well to ensure consistency in the working group approach and to achieve consensus. Other consensus processes will benefit from developing such a policy document.
Most working group members who were invited to serve readily agreed to do so. Finding dates for the first working group call and for the in-person working group meeting (which was mandatory to attend) proved to be challenging. Early in the process, one working group was delayed by 5 months, primarily due to scheduling problems. Because of that experience, the consensus coordinator placed more emphasis on the in-person meeting date during the initial conversation with working group experts.
Completing the working group deliberations in an 8-month timeframe (see Figure 3 ) also proved challenging. Because the first six working groups took an average of 12 months to complete, it became clear to the steering committee that they needed to make changes to complete the process on time. These changes included the following:
With these modifications, the last 13 working groups completed their activities in an average of 10 months.
Some working groups quickly applied the criteria for choosing measures and selected a set of measures that covered the scope of the domain. For other working groups, the measures selection process proved more challenging and required significant facilitation by the steering committee liaisons, working group supervisors, and working group managers. Beginning with the Environmental Exposures domain, working group members identified the protocols and made presentations recommending measures for consideration at the working group in-person meeting. Because these presentations were in the working group members’ areas of expertise, they were very effective in stimulating discussion, driving decisions, and selecting the measures.
The result of the consensus process is the PhenX Toolkit, a web-based catalog of 339 measures (295 measures in 21 domains and 44 measures in SAA collections; see Table 3) recommended by domain experts that is available for use at no cost (Hamilton et al., 2011). The purpose of this web-based resource is to facilitate the replication of the measurement protocols by investigators who are designing a new study or adding measures to existing human subjects research. The measures can be used in GWAS, clinical trial, case-control, observational, longitudinal, and gene-environment interaction (GxE) studies (Caspi et al., 2003). The Toolkit includes measures to explore environmental exposures that contribute to morbidity and mortality and measures that can be used to screen study participants for diseases, conditions, or exposures prior to including them in a study.
The Toolkit measures and protocols are being accessed in various ways (Hendershot et al., 2011). They are searchable by keyword and can also be browsed by domain of interest. To support browsing from additional perspectives, the steering committee approved the development of collections of measures. The collections, which are organized hierarchically, are (1) Health Conditions (e.g., pregnancy and birth [ Whitehead et al., 2012], lupus, osteoporosis, and skin cancer); (2) Risk Factors (e.g., diet and nutrition [ Stover, Harlan, Hammond, Hendershot, & Hamilton, 2010]; alcohol use and physical activity [ Haskell et al., 2012]); and (3) Substance Abuse and Addiction, as previously detailed. The mapping of PhenX measures to various standards (caDSR, common data elements, and LOINC) and to studies in dbGaP will help identify opportunities for cross-study analysis (Pan et al., 2012).
As reported in the PhenX newsletter, the top five most frequently accessed domains are Demographics; Alcohol, Tobacco and Other Substances; Anthropometrics; Cardiovascular; and Environmental Exposures (PhenX newletter, 2013, August 22). The top domains are calculated based on the number of times they are present in reports generated by users of the Toolkit. They, along with the top five measures, are listed on the home page of the PhenX Toolkit and are recalculated and updated with each new release.
The Toolkit has been cited in 41 Funding Opportunity Announcements from several government agencies, including NHGRI, NIDA, and other NIH ICs. As of May 2013, the Toolkit had received more than 417,500 visits from nearly 108,000 unique IP addresses. It has been used by people throughout the United States and in more than 151 countries. Because the Toolkit is designed to foster collaborations, more than 1,100 scientists have registered as Toolkit users. Investigators are asked to cite PhenX Toolkit measures in the studies being submitted to the dbGaP (www.ncbi.nlm.nih.gov/gap/) (Mailman et al., 2007), and PhenX has been referenced in 12 publications to date (see publications lists on the PhenX Toolkit website under the Resources tab).
NHGRI’s use of the Cooperative Agreement U series funding mechanism ensured collaboration between the project scientist and the RTI project team in developing the PhenX Toolkit, a catalogue of common measures for use in GWAS and other human subjects studies. The consensus process developed and managed by RTI and NHGRI enabled the scientific community to drive the content and features of the PhenX Toolkit. The outreach to the NIH ICs from the project’s inception and the engagement of the NIH liaisons in the steering committee and working group deliberations helped ensure that PhenX would benefit the broader scientific community and complement related research efforts. The success of PhenX depended on many stakeholders’ agreeing on what was important to measure and how to measure it.
Establishing a strong, diverse, broad-thinking, and interdisciplinary steering committee to provide guidance to the project was critical to the success of PhenX. The steering committee gave consistent leadership to the project. The collaborative relationship among the steering committee members and the RTI project team meant that the steering committee members could be tapped for advice in both formal and informal ways. The steering committee liaisons’ involvement in the working groups proved to be an effective way to structure the process and to obtain consistent results in spite of aggressive timelines. The steering committee set the initial scope of the domains and empowered the working groups to address that content. This resulted in different domains having different emphases. The Cancer domain, for example, is focused on preventive measures and risk factors, whereas the Cardiovascular domain concentrates on disease end points and other outcome measures. Having a PhenX Guidance Document for Working Group Members to capture the steering committee policy decisions and to set forth clear criteria for the choice of measures and protocols helped ensure uniformity in the working group approach.
With regard to the working groups, challenges included recruiting working group members who provided not only demographic diversity but also diversity of disciplines. Although PhenX sought senior scientists with stature in their fields, there was also an effort to recruit junior researchers who were emerging leaders in their research domains. During the recruitment process, the goal was to achieve a balance of academic, clinical, and federal researchers who came from different institutions and expressed diverse viewpoints.
With a shared vision of the task of prioritizing measures and selecting protocols, nearly all of the working groups achieved consensus through open discussion and reliance on working group members who were subject matter experts. The relatively small size of the working groups balanced the need for required expertise with the need to reach decisions by consensus. The leadership styles of the working group chair or co-chairs were essential to candid discussion.
The working groups typically relied on the expert opinions of their members and on working group deliberations to reach a consensus. In some instances, working group members were polled or asked to vote to prioritize the measures or to select specific protocols. Three working groups needed additional expertise to address the scope of the domain, and consultants were brought in to discuss the options and help with the deliberations.
The RFA suggested 15 to 20 research domains, with 10 to 15 measures for each domain. The goal was to keep the number of measures in the Toolkit relatively small, at least for the initial effort, and to demonstrate use of these measures by the scientific community. Another consideration was that investigators would probably not have the resources to add many measures to their studies, so it made sense to keep the resource concise and easy to navigate and to make the measurement protocols accessible to all investigators.
Weekly meetings between the project scientist and RTI staff facilitated the recruitment of NIH liaisons and the steering committee within the first 3 months of the project start date. The challenge was assembling a steering committee that could provide guidance to a project with such broad scope, ambitious goals, and tight timelines. If we had not succeeded in engaging a steering committee that could provide leadership in this trailblazing effort to stake out common measures for collaborative research, the success of this project would have been in jeopardy. Early in the process, the steering committee defined measure as a standardized way of capturing data on a certain characteristic of, or relating to, a study subject, and the steering committee encouraged working groups to use only well-established protocols.
Even so, the working groups struggled with criteria for selecting measures developed by the steering committee and the limited number of measures they were allowed to select to represent their research domain. The working groups also labored to limit the number of high-burden measures and to select measures that could be used by all investigators (i.e., measures that did not require a domain expert to interpret results).
In the working groups, there were some instances of advocacy for specific studies, protocols, or the preferences of specific professional associations. Although the steering committee indicated a clear preference for nonproprietary protocols, steering committee members were persuaded to allow proprietary protocols when they clearly provided the most accurate and reliable measurements. As a result, the PhenX Toolkit has a limited number of proprietary measures that require the investigator to go to the original source to obtain the measure.
The first three working groups were especially challenging because they were convened when the consensus process was still being developed, and there was no Toolkit to share as an example of how the selected measures would be presented. After the first few working groups and the initial release of measures in the Toolkit in February 2009, the process and the product were shared with new working groups, making the entire process easier. However, as more domains were completed, there was another challenge: how to effectively build on existing Toolkit content. For later working groups, during the introductory conference call, the principal investigator and project scientist emphasized the importance of reviewing measures already in the Toolkit. RTI staff informed working groups of existing Toolkit measures relevant to their research domain and of concurrent deliberations by working groups considering similar measures.
The RTI working group managers and supervisors played a critical role in bringing overlapping measures to the steering committee to address. Staff also facilitated discussions between the working group chairs and steering committee liaisons regarding the scope vis-à-vis the measures under consideration. The consistent advice from the steering committee was that if a measure was conceptually distinct and complementary to (and not competing with) measures already in the Toolkit, then it was a candidate for inclusion.
An important step in the consensus development process was to involve the scientific community in commenting on the proposed measures. The initial approach gave respondents the opportunity to review the protocols, prioritize the measures, and suggest additional measures and protocols. This outreach yielded a manageable number of comments with robust content that was of value to the working groups.
As subsequent domains elicited fewer responses, the steering committee decided to use an e-mail distribution list and to rely on the working group members to spread the word about the opportunity to comment. Some working groups were more active than others in identifying and encouraging their colleagues and the research community to review and comment on the selected measures. Although some domains received small numbers of responses, comments were always helpful to the working groups and were considered in the final deliberations. The change in approach proved more cost- and time-efficient and did not affect the quality of the input.
The PhenX team has identified the following components as key to an effective consensus process:
Driven by the scientific community and built using a consensus process, the PhenX Toolkit offers the research community common measures and protocols for collaborative research. This report describes the successful development and implementation of a consensus process for a complex project that relied on the participation of many scientists from many disciplines. The guidance provided by the steering committee and the expertise provided by the initial 21 working groups ensured that the content and features of the Toolkit meet the needs of the scientific community. The established consensus process was largely replicated for the SAA collections project, with its own SSP and three SAA working groups.
With the PhenX Toolkit, the NIH is “advancing a culture of scientific collaboration” (NIDA, 2012, page 1) by providing scientists with the necessary tools to enable productive collaborations now and in the future. The PhenX experience in reaching consensus on 339 measures provides a template for other collaborative efforts that would benefit from a consensus approach.
To ensure that the PhenX Toolkit remains relevant for future human subjects research, including genetic studies using next-generation sequencing, domains will be reviewed systematically. Under the direction of the steering committee, limited expansion of the Toolkit is envisioned, with new domains being considered and new measures being added. Future directions for the Toolkit include providing users with web-based protocols and with protocols that have been translated into languages other than English. Expert working groups will ensure that the content of the Toolkit continues to meet the needs of the scientific community and remains relevant to their research.
These terms and acronyms can also be found on the PhenX Toolkit website (www.phenxtoolkit.org) under the Resources tab.
| PhenX term | Definition |
|---|---|
| cancer Biomedical Informatics Grid (caBIG) | The caBIG is a collaborative initiative of the National Cancer Institute (NCI) that is developing standards-based infrastructure, tools, and data sets to support integrative cancer research and promote multidisciplinary scientific collaboration. More information: https://cabig.nci.nih.gov/ |
| cancer Data Standards Registry and Repository (caDSR) | The caDSR of the cancer Biomedical Informatics Grid (caBIG) is a database and set of Application Programming Interfaces (APIs) and tools to create, edit, control, deploy, and find Common Data Elements (CDEs) for use by metadata consumers and information about the Unified Modeling Language (UML) models and forms containing CDEs for use in software development. More information: https://cabig.nci.nih.gov/community/concepts/caDSR/ |
| Collection | A collection of measures with a shared characteristic, target population, or topic. The measures included in a collection may cut across research domains. |
| Common Data Element (CDE) | The purpose of defining CDEs is to represent unambiguously the data captured by a specific protocol. Each measure has a unique CDE created for its protocol. Each CDE is constructed by uniquely pairing the specific data element with its associated value. The PhenX CDEs are deposited at the cancer Data Standards Repository (caDSR). More information: https://cdebrowser.nci.nih.gov/CDEBrowser/ |
| Core collection | A Core collection of measures with a shared characteristic, target population, or topic that is fundamental to the study of substance abuse and addiction (SAA). These measures are appropriate for SAA and other researchers. |
| Data Collection Worksheet (DCW) | The DCW identifies each data item required by a protocol. The DCW helps investigators integrate selected PhenX measures into their data collection instrument. |
| Data Dictionary (DD) | The DD lists each variable in a protocol along with its attributes, including variable names and unique identifiers. The organization, content, and electronic format of the DD fully supports data submission to the database of Genotypes and Phenotypes (dbGaP). |
| database of Genotypes and Phenotypes (dbGaP) | The dbGaP presents results of genome studies and is maintained by the National Center for Biotechnology Information (NCBI). PhenX and dbGaP researchers collaborate with one another to provide more resources for the research community. More information: http://www.ncbi.nlm.nih.gov/gap/ |
| Domain | A domain is a field of research with a unifying theme and easily enumerated quantitative and qualitative measures (e.g., demographics, anthropometrics, organ systems, complex diseases, and lifestyle factors). |
| Essential measure | An essential measure is another PhenX Toolkit measure that is needed to interpret the results of the measure of interest, such as age when measuring height or weight. |
| Genome-wide association studies (GWAS) | GWAS are an approach used in genetics research to associate specific genetic variations with particular diseases. The method involves scanning the genomes of many different people and looking for genetic markers that can be used to predict the presence of a disease. Once such genetic markers are identified, they can be used to understand how genes contribute to the disease and develop better prevention and treatment strategies. More information: http://www.genome.gov/Glossary/index.cfm?id=91 |
| Logical Observation Identifiers Names and Codes (LOINC) | LOINC is a set of data standards for identifying, exchanging, and pooling clinical information for clinical care, outcomes management, and research. The LOINC database provides a set of universal names and ID codes for identifying laboratory and clinical test results in the context of existing HL7, ASTM E1238, and CEN TC251 observation report messages. More information: http://loinc.org/ |
| Measure | A measure refers broadly to a standardized way of capturing data on a certain characteristic of, or relating to, a study subject. |
| National Human Genome Research Institute (NHGRI) | NHGRI is the National Institutes of Health (NIH) Institute that provides funding for PhenX through a cooperative agreement (1U01 HG004597-01). More information: http://www.genome.gov/ |
| National Institute on Drug Abuse (NIDA) | NIDA is a National Institutes of Health (NIH) Institute that provided additional funding for the PhenX Substance Abuse and Addiction (SAA) project through an administrative supplemental to the National Human Genome Research Institute (NHGRI) cooperative agreement (U01 HG004597). More information: http://www.drugabuse.gov/ |
| National Institutes of Health (NIH) | NIH is a federal agency whose mission is to improve the health of the people of the United States. NIH is a part of the Public Health Service, which is part of the US Department of Health and Human Services. The NIH is made up of 27 Institutes and Centers (ICs), each of which has its own specific research agenda. |
| NIH Institutes & Centers (ICs) Liaisons | Liaisons from the ICs are invited to participate in PhenX to lend their substantial expertise and experience to the PhenX consensus-building process. More information: https://www.phenx.org/Default.aspx?tabid=66 |
| Office of Behavioral and Social Sciences Research (OBSSR) | OBSSR, in the National Institutes of Health (NIH) Office of the Director, provided additional funding for PhenX through administrative supplements to the National Human Genome Research Institute (NHGRI) cooperative agreement (U01 HG004597). More information: http://obssr.od.nih.gov/index.aspx |
| Patient-Reported Outcomes Measurement Information System (PROMIS) | PROMIS, funded by the National Institutes of Health (NIH), is a system of measures of patient-reported health status for physical, mental, and social well-being. PROMIS tools measure what patients are able to do and how they feel. PROMIS measures can be used as primary or secondary end points in clinical studies of the effectiveness of treatment. More information: http://www.nihpromis.org/ |
| PhenX | PhenX (“consensus measures for Phenotypes and eXposures”) is a cooperative agreement (1U01 HG004597-01) funded by the National Human Genome Research Institute (NHGRI). The project is led and managed by a team of RTI scientists to develop the PhenX Toolkit. |
| PhenX Toolkit | The PhenX Toolkit is a web-based catalog of high-priority measures for consideration and inclusion in genome-wide association studies (GWAS) and other human subjects research efforts. For each measure, the PhenX Toolkit provides standard protocols to collect the measures, Common Data Elements (CDEs), and supporting documentation. More information: https://www.phenxtoolkit.org/ |
| Protoco | A protocol is a standard procedure recommended by a Working Group for investigators to collect and record a measure. |
| Related measure | A related measure may be helpful, based on the selection of another measure. Related measures may be needed to calculate a derived variable (e.g., Weight and Height are needed to calculate body mass index [BMI]), commonly used to present the value of the chosen measures (e.g., Height and Weight by Race and Ethnicity), conceptually related to the chosen measure (e.g., Current Educational Attainment is conceptually related to Annual Family Income), or physiologically and/or biologically related to the chosen measure (e.g., Stroke and Heart Disease). Related measures are suggestions only. |
| Request for application (RFA) | An RFA is a formal statement that solicits grant or cooperative agreement applications in a well-defined scientific area to accomplish specific program objectives. Source: http://grants.nih.gov/grants/glossary.htm#R11 |
| RTI International (RTI) | The Research Triangle Institute, also known as “RTI International” or “RTI,” is an independent nonprofit research institute dedicated to improving the human condition. PhenX is led and managed by a team of RTI scientists. More information: http://www.rti.org/ |
| Specialty area | A Specialty area is a field of Substance Abuse and Addiction (SAA) research with a unifying theme and easily enumerated quantitative and qualitative measures related specifically to risk factors, community substance abuse, and addiction. |
| Specialty collection | A Specialty collection of measures with a shared characteristic, target population, or topic that is related to Substance Abuse and Addiction (SAA). The use of these measures is study specific. |
| Steering Committee (SC) | The SC is responsible for prioritizing research domains and providing overall guidance to the PhenX project. More information: https://www.phenx.org/Default.aspx?tabid=57 |
| Substance Abuse and Addiction (SAA) project | The SAA project expands the breadth and depth of SAA-related measures in the PhenX Toolkit by adding six Specialty collections and one Core collection of SAA measures. |
| Substance Abuse and Addiction Scientific Panel (SSP) | The SSP, part of the SAA project, is responsible for selecting the Specialty areas, approving the Specialty collections, and selecting the Core collection. |
| Substance Abuse and Addiction Working Group (SAA WG) | An SAA WG addresses two Specialty areas. Each WG includes 7–8 scientists with relevant expertise. The SAA WG is responsible for recommending measures for the PhenX Toolkit. |
| Supplemental Information (SI) | SI describes the scope of each domain, includes other measures considered by the Working Group (not selected for the PhenX Toolkit), and has additional information. More information: https://www.phenx.org/Default.aspx?tabid=281 |
| Working Group (WG) | A WG includes 6–8 scientists with relevant expertise in each research domain. The WG is responsible for recommending measures for the PhenX Toolkit. More information: https://www.phenx.org/Default.aspx?tabid=58 |
Deborah R. Maiese, MPA, is the PhenX consensus coordinator and serves as a consultant to the Research Computing Division at RTI International.
Tabitha P. Hendershot, BA, the PhenX co-investigator, is a research public health analyst in the Health Sciences Division at RTI.
Lisa C. Strader, MPH, a former PhenX co-investigator, is a senior research epidemiologist and manager of RTI’s Global Epidemiology and Health Research Program.
Diane K. Wagener, PhD, a former PhenX co-investigator, is a senior research epidemiologist in the Genomics, Statistical Genetics, and Environmental Research Program at RTI.
Jane A. Hammond, PhD, is a PhenX working group supervisor and a senior research psychologist in the Health Sciences Division at RTI.
Wayne Huggins, PhD, is a PhenX working group supervisor and a research programmer and analyst in the Research Computing Division at RTI.
Richard K. Kwok, PhD, a former PhenX working group supervisor, is a staff scientist in the Epidemiology Branch at the National Institute of Environmental Health Sciences, National Institutes of Health (NIH).
Dana B. Hancock, PhD, is a PhenX working group supervisor and a genetic epidemiologist in the Behavioral Health Epidemiology Program at RTI.
Nedra S. Whitehead, PhD, is a member of the PhenX Toolkit development team and a senior genetic epidemiologist in RTI’s Genomics, Statistical Genetics, and Environmental Research Program.
Destiney S. Nettles, BA, is a PhenX project administrator, PhenX working group manager, and project administration specialist in the Molecular Epidemiology, Genomics, Environment and Health Division at RTI.
Joseph G. Pratt, MPM, is the PhenX project manager and associate director of Molecular Epidemiology, Genomics, Environment and Health Division at RTI.
Marcia S. Scott, PhD, is the PhenX liaison from National Institute on Alcohol Abuse and Alcoholism (NIAAA), NIH, and is a program director in the Division of Epidemiology and Prevention Research at NIAAA.
Kevin P. Conway, PhD, is the PhenX liaison from National Institute on Drug Abuse (NIDA), NIH, and is Deputy Director of the Division of Epidemiology and Prevention Research at NIDA.
Heather A. Junkins, MS, is a health science analyst in the Division of Genomic Medicine, National Human Genome Research Institute (NHGRI), NIH, Bethesda, Maryland.
Erin M. Ramos, PhD, is the PhenX project scientist in the Division of Genomic Medicine, NHGRI, NIH, Bethesda, Maryland.
Carol M. Hamilton, PhD, is the PhenX principal investigator and Director of the Bioinformatics Program, Research Computing Division, at RTI.
RTI International is an independent, nonprofit research organization dedicated to improving the human condition. The RTI Press mission is to disseminate information about RTI research, analytic tools, and technical expertise to a national and international audience. RTI Press publications are peer-reviewed by at least two independent substantive experts and one or more Press editors.
Maiese, D. R., Hendershot, T. P., Strader. L. C., Wagener, D. K., Hammond, J. A., Huggins, W., Kwok, R. K., Hancock, D. B., Whitehead, N. S., Nettles, D. S., Pratt, J. G., Scott, M. S., Conway, K. P., Junkins, H. A., Ramos, E. M., and Hamilton, C. M. (2013). PhenX—establishing a consensus process to select common measures for collaborative research.RTI Press Publication No. MR-0027-1310. Research Triangle Park, NC: RTI Press. https://doi.org/10.3768/rtipress.2013.mr.0027.1310
RTI International is an independent, nonprofit research institute dedicated to improving the human condition. We combine scientific rigor and technical expertise in social and laboratory sciences, engineering, and international development to deliver solutions to the critical needs of clients worldwide.
This work is distributed under the terms of a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 license (CC BY-NC-ND), a copy of which is available at https://creativecommons.org/licenses/by-nc-nd/4.0/legalcode
Your browsing activity is empty.
Activity recording is turned off.
See more...
