Simpler surveys of whether and how data are used by policy makers may be the only realistic option

a tool for ex-post evaluation with good discriminating ability that makes it possible to measure the extent to which a project’s objectives have been met . . . and the extent to which the project contributed benefits in terms of impact on the group’s scientific performance and social payback

A shared and comprehensive conceptual framework does not seem to be available yet and its single components (epidemiological, economic, and social) are often valued differently in different models

CUSP for CLABSI [central line-associated bloodstream infections] achieved a successful national roll-out and significant results, reducing CLABSI rates by 40% from baseline

There were numerous examples of work conducted by former FIC trainees that influenced national and global policies

Facilitators for this influence included . . . professional networks spanning research and policy communities

The approach selected to evaluate the impact of research programmes on policy also needs to be sensitive to the context in which the evaluation will be undertaken

All these experiences together with our own work suggest that the impact of HTA on practices and introduction of new technologies is higher the more circumscribed is the target of the recommendation

to evaluate ‘impact’ (the extent to which change occurred)

allowed for the accumulation of evidence in addition to the development of collaborative ties between researchers and practitioners, which ultimately resulted in changes to the service delivery environment

a profound impact on national health policy, donor priorities and public health action

In particular, the choice of an evaluation objective, we find, is immensely important. It, directly or indirectly, influences the appropriateness of all other key elements

The reviews had influenced the development of national and international policy, although much of the impact was at a ‘micro’ level in the form of practice guidelines

Although there were examples of policy documents being informed by health visiting research it was not always clear what role research had played in the development of recommendations

Results from the semi-structured interviews suggest that searching for relevant Cochrane reviews is part of the guideline development process

more work is needed to develop suitable methods for evaluating the impact of systematic reviews

It is essential that academic health centres engage actively in ensuring that a culture of research inquiry is maintained and that support is available to those researchers that may ultimately contribute to the excellence in health care

changes to practice, changes to services, policy change, commercialisation

We found that single intervention research studies can and do have concrete and measurable post-research real-world impacts . . . on policy and practice

Our eventual goal is to develop an aggregate composite score for measuring impact attainment across sites

A statement of claims against impact criteria, up to four case studies illustrating examples of impact, and details of end users who can verify the impact claims

apply their collective expert judgement to determine the validity of the claims made against the impact criteria. Impact ratings will be assigned . . . The Working Group recommended the Payback consensus scoring approach as particularly suited for this purpose

an innovative, Web-based application intended to capture and track short- and long-term research outputs and impacts

the lack of direct attribution of NIEHS-supported work to many of the outcome measures and the lack of robust electronic databases that can be easily searched to help establish these linkages

Distinctions drawn between the institutional pathways are artificial to some degree, and there is considerable cross-over between sub-models

The main impression, therefore, is that the systems for protection against injury described above provide major economic benefits

Extensive follow-up of the post-project impact of completed projects was not possible . . . we were unable to determine whether project deliverables were accessed and used by the relevant end-users, nor could we examine possible effects on population health

advancing knowledge, capacity building, informing decision-making [changed from the original informing policy and product development]; health benefits; and broad economic and social benefits [changed from broader economic benefits]’

recommend a method that builds on the advantages of the ‘payback model’ but adapts it to target-specific impacts in multiple domains at multiple levels

One is to distinguish conceptually between output and impact (or outcome). The second is to identify specifically the transfer mechanisms by which science translates into impact. The last is to develop appropriate and reliable instruments and indicators

(1) a performance measurement system that tracks progress to impact; (2) aggregated and pooled reporting capabilities through the standardization of indicators and metrics across programs

adoption of additional impact categories, indicators, and measures which improved the organization’s ability to assess and demonstrate its contributions to health system impacts in addition to the contribution of its grantees

High quality research carried out in Australian universities has had enormous benefits for the health, security, prosperity, cultural and environmental wellbeing of Australia, the region and the world

Structured information is intended to facilitate and underpin the decision-making of EC officers in charge of project management, and support them in the design of future research topics and programmes

results from case studies revealed a high concordance with the coordinators’ survey on several facets, for instance, in . . . providing evidence of project performance, and revealing some types of impacts

the study design is descriptive rather than quasi-experimental, and therefore does not include a comparison group composed of other RWJF national programs

some approaches utilised by the program worked well, including developing a multifaceted, ongoing, interactive relationship with advocacy and policy-maker organisations

ALR was the catalyst to build a new field of transdisciplinary research focused on policy and environmental factors conducive to physical activity

This aligns with the paper by Hage (appendix A, p. A79), which argues that meso-level factors—those at the health category level—are vital in understanding the impacts and pathways to impacts of health research

CAHS⁷

our initial planning and mapping exercise explored several major theoretical approaches that could, potentially, contribute to informing the conduct of the review and to building a framework within which to analyse the mechanisms through which engagement in research can improve performance

We have called these two dimensions the degree of intentionality and the scope of the impact

important to attempt to make such categorisations because of the potentially very different mechanisms that may be at work in these different circumstances on the two dimensions

it may not be necessary to develop another version, but rather find clever ways to combine elements from different frameworks to best fit the particularities of a certain research topic

Research teams who started the policy dialogue early and maintained it throughout the study, and teams that engaged with decision-makers at local level, district and national levels simultaneously were more successful in translating research results into policy action

. . . the uptake of generated evidence in national health policy or international policy guidelines. Because of the interactive nature of AHSI-RES significant results have already been achieved, but the policy dialogue is not yet complete and further uptake can be anticipated

. . . benefits to one or more areas of the economy, society, culture, public policy and services, health, production, environment, international development or quality of life, whether locally, regionally, nationally or internationally

. . . manifested in a wide variety of ways including, but not limited to: the many types of beneficiary (individuals, organisations, communities, regions and other entities); impacts on products, processes, behaviours, policies, practices; and avoidance of harm or the waste of resources

By inference it must be responsible for a significant part of the improvement in health outcomes in people with Type 2 diabetes over the last decade

Assessing impact of a substantial number of projects would be more feasible if the burden of response could be reduced by refining and streamlining the methods

Research does not work like a cannon shooting knowledge into the world of action, where the targeting and force of the knowledge determines its ‘impact’.

Instead, the productivity of research for health ultimately lies with the users who have to pick up and combine knowledges (in the plural), using them for their own purposes . . .

The consequence is that achieved changes cannot realistically be attributed to a single research project as ‘source’

We found that there is great potential for national and international guidelines to be used as sources of information to help further our understanding on the impact of research on practice: the challenge is to be able to harness that information in an efficient way

designed as a ‘DIY’ approach with descriptive categories that prompt researchers to systematically think through and describe the impact of their work.

While initially sceptical, LSHTM researchers found [it] prompted them to identify a wide range of impacts related to their work in a relatively systematic manner (compared to the ad hoc approaches they had previously used)

Beyond publications, indicators of other activities, outputs, and outcomes are not as well supported

Researchers and policy-makers come from different worlds and do not understand each other. There is much evidence that effective knowledge exchange occurs through sustained linkage and dialogue

a growing body of empirical work demonstrates that those benefits are substantial

study will employ a novel method of participatory, utilisation-focused evaluation

(knowledge user and researcher partnerships established; knowledge generated; relevant research results are disseminated and/or applied by partners and knowledge users; advancement of knowledge translation science)

Intermediate outcomes (knowledge users and researchers learn from each other; knowledge users are informed by relevant research; generalisable knowledge is created and disseminated)

Flows of knowledge, expertise and influence

no evident contradictions between results obtained by different methods. However, the level of detail provided did vary; unsurprisingly, some methods, such as the survey, enabled us to achieve greater breadth of data while others, such as the case study interviews, enabled us to probe issues in more depth

meant there was no special research orientation towards users or ‘relevant themes’

Changes in practice or policy often appear to stem from a general ‘awareness-raising’ or conceptual shift. Precise measurement of the impact of research upon a particular change in practice is likely to be unattainable

2,879 reports of policy influences between 2006 and 2012 . . . in 1,083 awards

The method consisted of a process to get a societal relevance score per research department based on its (research) outreach to relevant societal stakeholders. These quantitative scores were then compared to standardised scientific quality scores (CWTS indicator) based on scientific publications and citations of peer-reviewed articles

the nature and quality of the intervention itself . . . high quality research, champions who advocated for adoption, and active dissemination strategies. Our findings also highlight the need for strong partnerships between researchers and policy makers/practitioners to increase ownership over the findings and commitment to action

A total of 16 different impact assessment models were identified, with the ‘payback model’ being the most frequently used conceptual framework

participants inside and outside of state government have used the [CHBRP] . . . reports as both guidance in policy design and as political ammunition . . . Almost every respondent noted that CHBRP reports, however authoritative, served as ‘political ammunition’ in the manner described by Weiss (1989)

this method does not support attribution of these outcomes to specific research activities nor to specific funding sources. Nor did we gain many insights from the survey into the pathways by which research is translated and used to affect these outcomes

the model guiding this assessment, with its “pathways”, is a reasonable representation of how research may result in such impacts, at least as they pertain to asthma research

Assessing impact is feasible

While the majority of the proposed indicators have some validity, there are significant challenges in refining the current indicators and methodologies in a number of ways

an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia

While aspects of impact can be adequately interpreted using metrics, narratives, and other evidence, the mixed-method case study approach is an excellent means of pulling all available information, data, and evidence together, allowing a comprehensive summary of the impact within context

to remain active in the project, when their timelines tend to be more short-term

all but one team felt the scientific rigor of the projects had improved as a result of the extended planning and interaction with decision-makers as well

the interaction has led to use of data in decision-making, as well as a clear recognition by both parties that different kinds of evidence were at play

Research questions come from primary stakeholders. There is involvement from these stakeholders in agenda and priority setting. The quality of KCE research itself is high and in general beyond discussion. The relevance of KCE research findings is generally judged as high

In an earlier study we collected qualitative data through interviews with CIs and gathered copious information that provided more context to better understand the results

if HEIs were proactive in collecting more qualitative and quantitative data evidencing the reach and significance of the impact. International MPA members cautioned against attempts to ‘metricise’ the evaluation of the many superb and well-told narrations describing the evolution of basic discovery to health, economic and societal impact

p. 11

believes that the collection of impact case studies provide a unique and powerful illustration of the outstanding contribution that research in the fields covered by this panel is making to health, wellbeing, wealth creation and society within and beyond the UK

the boldest, largest and most comprehensive exercise of its kind of any country’s assessment of its science . . . to our knowledge, [it is] the first systematic and extensive evaluation of research impact on a national level. We applaud this initiative by which impact, with its various elements, has received considerable emphasis

clear and compelling narrative linking the research programme to the claimed impact; verifiable evidence (qualitative or quantitative) to support the claimed impact provided within the text . . . and (where appropriate) spread of the impact beyond the immediate beneficiaries to a much broader and possibly global audience

there is great value in comparing and contrasting our own ‘insider-outsider’ perspectives with those of pure “insiders and outsiders” as this has helped to clarify the reasons behind differences of opinion

triangulating information from the interviews, earlier evaluations, and external review reports

At CHP, discussing research ideas and the possible implementation of research findings with policy makers at the start-up of projects, and then presenting findings face-to-face, became a powerful mode of influencing policy

examination of the use of research in policy documents (which speaks to research utilisation theory); use of research in policy processes (which speaks to interactive engagement theories) and dissemination of research in the public sphere via the media

The approach is less complicated than others that have been suggested (Donovan and Hanney;⁴⁵ Hanney et al., 2003;³⁵ Lavis et al., 2003³⁵⁴) but goes beyond a simple checklist approach (Smith, 2001), whilst also being grounded in policy theory

should consider carefully the correspondence of the indicators with the agency’s implementation logic model

centers will be more receptive to the evaluation process if it is clear how the data are being used and if the information is considered in the context of how centers are designed and operate

Important to note, was the extent to which respondents highlighted this as being in the context of the participating services or host organisations

Specific databases and resources for each indicator are identified and search tips are provided’. However, in relation to their pilot case study state; ‘For some indicators, supporting documentation was not publicly available. In these instances, contact with policy makers or other officials was required

may provide a tool for research investigators not only for documenting and quantifying research impact, but also . . . noting potential areas of anticipated impacts for funding agencies

examine connections between the generation and communication of basic and clinical knowledge and the impact of this knowledge along different translational pathways

NIH’s data infrastructure was built primarily to manage grants and contracts during their life cycle, not to track outcomes

teams that include decision-makers or users of health of health information were more effective in achieving outcomes in health policy or practice from the research findings

The REPP profiles represent the various existing constellations of interactions between a program and its environment. Because of the large quantity of indicators, it is not always easy to transform them interpretatively in the profile to construct such a constellation

Social impact of scientific research refers to measurable effects of the work of a research group or program or a research funding instrument in a relevant social domain. The effect regards the human well-being (‘quality of life’) and or the social relations between people or organisations

wide range of research [publication], capacity building and knowledge translation outputs

introduces two new indicators of a centre’s impact, namely the percentages of their papers that are cited on a set of 43 UK cancer clinical guidelines and in stories appearing on the BBC website

We have found substantial variation in the propensity of papers originating from UK cancer centres to be cited on guidelines and in the media. There does not appear to be a correlation with the conventional citation impact of the papers, nor indeed with the size of the centre

As a conceptual framework, the Payback Framework drew attention to a range of outcomes less often considered in other NIH program evaluations and ordered these in a logical manner

working in partnership with all relevant stakeholders to refine and develop research agendas and undertake research builds an appreciation of research that intensifies the effectiveness of the research to policy and practice interface

Improvements in health care (e.g. improved surgical treatment of brain tumours through pre-op MRI and intra-op ultrasound)

TTURC researchers report considerable impact on policies at the state and local levels and on practice with respect to tobacco control

optimism that their research will help lead to significant positive health outcomes, especially for consumption and prevalence [however] they reflect researcher attitudes but do not constitute evidence of long-term impact

Research funders should identify and fund further research into evaluation methods with a rigour and quality equivalent to other research fields . . . UK research funders should support research to assess the economic impact of UK medical research, which should include critiques of existing economic approaches

by rewarding the process of knowledge exchange, rather than specific impacts, this approach would incentivize participation by a broader range of academics

evidence on the varied types of knowledge exchange activity undertaken by academics in different disciplines and institutional types calls into question the likely effectiveness of knowledge exchange drivers based on outcomes

The Assessment Framework Report predominantly describes outputs and achievements associated with Trust activities though, where appropriate, inputs are also included where considered a major Indicator of Progress

Case studies and stories have gained increasing currency as tools to support impact evaluation, and are a core component of institution submissions within the UK Research Excellence Framework

In future years, as the Trust further integrates its online grant progress reporting system throughout its funding activities (through Wellcome Trust e-Val and MInet) it will be easier to provide access to, and updates on grant-associated outputs throughout their lifecycle

economic analysis is highly integrated into the decision-making process of NICE’s technology appraisal programme

This reversal of order moves beyond the program theory articulated in the logic model and essentially places the burden on research programs to trace backward how specific outcomes were generated from research activities

reduced work-related hazardous exposures or reductions in occupational injuries, illnesses, and fatalities within a particular disease- or injury-specific area

The most significant aspect of program effectiveness is impact- the outcome of the program, which otherwise would not have occurred without the program intervention

Overall the indications are that the instrument is an effective and low burden way of collecting an overview of the impacts arising from a portfolio of research . . . [But] depends on the researchers’ knowledge and honesty, and simplifies quantification of impacts and their attribution

It appears that these comparative effectiveness reviews by the AHRQ have some indirect impact on formulary access in the US. However, price and contracting, in addition to efficacy and safety are among the key determinants for plans

participant selection that favours programme staff, studies being focused on healthcare delivery; research questions being of direct programme relevance; early engagement and buy-in of programme managers and/or policy makers; and inclusion of stakeholders as co-authors on publications

they had at least a selective effect on volumes supplied, expenditure and/or resource distribution