U.S. flag

An official website of the United States government

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Rentel MC, Simpson K, Davé A, et al. A 10-year impact assessment of the Efficacy and Mechanism Evaluation (EME) programme: an independent mixed-method evaluation study. Southampton (UK): NIHR Journals Library; 2021 Nov. (Efficacy and Mechanism Evaluation, No. 8.20.)

Cover of A 10-year impact assessment of the Efficacy and Mechanism Evaluation (EME) programme: an independent mixed-method evaluation study

A 10-year impact assessment of the Efficacy and Mechanism Evaluation (EME) programme: an independent mixed-method evaluation study.

Show details

Appendix 2Survey and interview response and questionnaires

Response rates and characteristics

Of the 91 CIs of active EME projects contacted, 46 (51%) responded to the survey. A comparison between respondents and non-respondents was performed to verify that the survey sample was representative of the overall population. No major differences between the projects led by respondents and non-respondents in the distribution of award year, size, workstream and HRCS categories were observed.

Of the 93 CIs of unsuccessful applications contacted, 28 (30%) responded to the survey.

Interviews were conducted with 51% (23/45) of CIs contacted whose projects were either completed or close to completion. Only four of these awards were funded through the commissioned funding stream and the remainder were researcher-led applications. Other project characteristics did not show major differences between respondents and non-respondents (e.g. number of respondents vs. non-respondents per award start year or average size of award). Interviews were conducted with CIs across each of the HRCS health categories, with the exception of ‘renal and urogenital’, ‘neurological’, ‘musculoskeletal’ and ‘injuries and accidents’.

Survey questionnaire: active projects

About your EME project.

Please confirm the EME grant number that relates to this survey:

Research activity

1.

Where did the original idea for the research topic come from? Select all that apply:

  • Academic researchers.
  • Clinical researchers.
  • Patient groups.
  • Health professionals.
  • Regulators.
  • Industry partner.
  • Literature.
  • Other.

Please provide further details.

2.

How was the preceding (proof-of-concept) study funded? Select all that apply:

  • Funding from NIHR.
  • Funding from MRC.
  • Funding internal to my organisation.
  • Industry.
  • Charity.
  • Other.

Please specify source of funding and funding stream (if applicable).

3.

Had you worked with the research team for this project before? [Select one.]

  • Yes.
  • No.

If no, how did you identify the co-investigator(s) and why did you include them in the team?

4.

Does/did the project involve partnership(s) with industry, charities, (other) health-care professionals and/or international academic partners? [Select one.]

  • Yes.
  • No.

If yes, who are the partners and what do they bring to the project?

5.

Is/was your grant co-funded from other sources? [Select one.]

  • Yes.
  • No.

If yes, please provide a brief description of the origin and nature of support received and the role that this has played in enabling the research project.

6.

If your project does/did not receive industry partnership funding, had you sought industry co-funding or in-kind contribution? [Select one.]

  • Yes.
  • No.
  • N/A (we received industry contribution).

If yes, please explain why industry contribution was not provided.

If no, please explain why industry contribution was not sought.

7.

What research infrastructure is/was the project drawing on? Select all that apply:

  • Biomedical Research Centre (BRC).
  • Biomedical Research Unit (BRU).
  • Clinical Research Network (CRN).
  • NHS resource.
  • MRC-supported research infrastructure.
  • Charity-supported research infrastructure.
  • Industry research infrastructure.
  • Other (please specify).

Please provide a brief description of your answer.

8.

In the light of your experiences to date, does/did the project have the right partners and infrastructure? [Select one.]

  • The expertise and infrastructure accessed by the project is/was appropriate.
  • Critical expertise is/was missing.
  • Critical infrastructure is/was missing.
  • Critical infrastructure and expertise are/were missing.

Please provide further details.

9.

Does the project involve any of the following aspects? Select all that apply:

  • Mechanistic study.
  • Repurposing of an existing intervention.
  • Applying novel study design.
  • Generating/use of digital health data.
  • Precision medicine approaches.
  • Regenerative medicine approaches.
  • Including comorbidities and multimorbidities.
  • Targeting age-related condition.

Please provide further details.

Patient and public involvement

10.

How many patient and public representatives (PPI) are/were involved in the project? [Select one per row.]

One or two people are/were involvedThree or more people are/were involvedNo patient or public representatives involved
In the design phase (before submitting the application)
In the delivery phase

Please provide further details.

11.

If relevant, what was the frequency of patient and public involvement? [Select one per row.]

OnceOn an ad hoc basisContinuously
In the design phase (before submitting the application)
In the delivery phase

Please provide further details.

12.

If patients/members of the public are/were involved in the design and/or delivery of the project, which task(s) did they contribute to? Select all that apply:

  • Developing the research questions.
  • Developing the research methodology and approach.
  • Design of the recruitment or retention strategy.
  • Development of patient-facing information.
  • Directly approaching/recruiting or retaining study participants.
  • Dissemination of information and study findings.
  • Other (please specify).

Please describe task(s) in more detail.

13.

In your view, to what extent is/was patient and public involvement (PPI) an important enabler for the project? [Select one per row.]

To a large extentTo a limited extentNot at allDo not know
In the design phase
In the delivery phase
In supporting future use of project findings

Please explain your answer.

Challenges encountered and adjustments to project plan.

14.

Did you face any of the following challenges in the implementation of the research project? Select all that apply:

  • No challenges encountered.
  • Contracting processes.
  • Difficulty securing ethics approval in the time frame required.
  • Setting up study sites.
  • Capacity issues/shortage of trained staff.
  • Patient recruitment.
  • Supplier issues.
  • Lack of infrastructure/equipment.
  • Other resource issues.
  • Methodology issues (e.g. related to statistics, data return, loss to follow-up).
  • Regulatory issues.
  • The intervention was not safe.
  • Change in guidelines/practice during research project.
  • Other (please specify).

Please describe the main challenges encountered.

15.

Did you have to adjust any of the following aspects of the project plan after the start of the project? This question does not apply to adaptive trials. Select all that apply:

  • None, the project plan has aligned closely with plan put forward in the proposal.
  • Patient recruitment target number.
  • Study timeline.
  • Type of data collected/outcome measures.
  • Method of data collection.
  • The study team.
  • Level/frequency/nature of patient and public involvement.
  • The mechanistic study/explanatory aspects of the project.
  • Other (please specify).

If changes were made, please describe these and explain how they have helped to address the challenges encountered.

16.

In the light of your experiences to date, would you approach the project’s design or implementation differently? [Select one.]

  • No, I would not make any changes to the project’s design and implementation.
  • Yes, I would make minor changes to the project’s design and implementation.
  • Yes, I would make substantial changes to the project’s design and implementation.

If yes, which aspects would you change? Select all that apply:

  • Patient recruitment target number.
  • Study timeline.
  • Preparatory data collection (e.g. in a feasibility study).
  • Type of data collected/outcomes measures.
  • Method of data collection.
  • Recruitment of additional experts to the team.
  • Level/frequency/nature of patient and public involvement.
  • The mechanistic study/explanatory aspects addressed by the project.
  • Other (please specify).

Please outline the main changes you would make and why.

Key findings and project outputs

Research findings

17.

What findings, if any (including emerging and unpublished), has your research project resulted in to date? Select all that apply:

  • No findings to date: the project is ongoing.
  • Findings related to efficacy demonstrated positive benefits through the primary outcome.
  • Findings related to efficacy demonstrated no benefit through the primary outcome.
  • Findings related to causes of differences in response to the intervention in different patient groups.
  • Findings related to the mechanism of action of the intervention.
  • Findings related to disease mechanisms.
  • Other findings.

Please provide a brief summary of key findings.

Did the findings allow a decision on whether to take the research further? [Select one.]

  • Yes.
  • Not yet.
  • No.

If no, why not?

Skills and knowledge

18.

Has the EME-funded research contributed to training and/or capacity building? Select all that apply:

  • Not relevant.
  • Not yet.
  • Yes: clinical research fellows.
  • Yes: staff members.
  • Yes: other trainees.

If yes, please describe the skills and knowledge developed (e.g. disease-/intervention-specific knowledge, translational research methods, regulation, industry standards, patient involvement, team working).

Project outcomes

Uptake of project findings by research community

19.

Have others taken up project findings or been using new tools, databases, banked samples or methodologies developed as part of the EME-funded project? [Select one.]

  • Do not know.
  • Not yet.
  • Yes.

If yes, please provide further detail.

Further funding and collaboration

20.

If you are planning further research based on the EME-funded project, what sources of funding do you intend to apply to? Select all that apply:

  • NIHR.
  • MRC.
  • Charity.
  • Industry.
  • Other.
  • No further research planned.
  • Further funding already secured.

Please provide details.

21.

Has the project led to further collaboration as a result of the EME-funded project?

Yes, we are collaborating in a new projectNot yet, but planning to do so in the futureNo, I do not expect future collaborations
With existing team members
With existing industry partners
With new collaborators

Please explain your answer.

22.

Has the EME-funded work led to public health benefits? [Select one.]

  • Not relevant.
  • Not yet: the project is ongoing.
  • Not yet, but it has the potential to do so.
  • Yes: the EME-funded work has informed clinical guidelines or practice.
  • Yes: other.

If yes, please provide further detail.

23.

Has the project led to any commercial benefits, e.g. for the academic sector, industry partners or others? [Select one.]

  • Not relevant.
  • Not yet: the project is ongoing.
  • Not yet, but it has the potential to do so.
  • Yes: the EME-funded work has led to commercial benefits.

If yes, please provide further detail.

24.

Please describe any other outcomes, including unintended outcomes, of the EME-funded work.

Funding landscape

25.

Do you think the EME programme fills a gap in the funding landscape, enabling progress of interventions along the translational pathway? [Select one.]

  • Yes, the EME programme fills a gap in the funding landscape.
  • To a limited extent, the EME programme contributes to progressing interventions but gaps remain.
  • No, the EME programme does not fill a gap.
  • Do not know.

Please explain your answer.

26.

What funding programme could you have applied to if the EME scheme had not been available?

27.

What sources of further funding are available to continue to develop a promising intervention following on from an EME grant?

Design and management of the EME programme

28.

Based on your experience, do you consider the EME programme appropriate in terms of funding amount, project length and scope? [Select one.]

  • Yes, the EME programme is appropriate for achieving its aim.
  • No, the EME programme is not appropriate and it needs to change.
  • Do not know.

Please explain your response.

29.

What do you consider the main strengths of the EME programme? What sets it apart from other related funding programmes?

30.

What do you consider the main weaknesses of the EME-programme? What improvements would you suggest?

31.

Should the EME programme support additional activities to make the research more effective and increase its potential for impact? [Select one.]

  • Yes.
  • No.
  • Do not know.

Please explain what additional activities should be supported and why.

32.

Was the management and support provided by the EME programme staff and EME Board (Review Committee) helpful for you (e.g. the call guidelines; communication with the programme team; knowledge and support provided by the EME Board as part of the review process; interactions and advice as part of project monitoring and reporting)?

YesNoDo not know
EME programme staff: at proposal stage
EME Board: at proposal stage
EME programme staff: at implementation stage

Please provide further details.

Final comments and close

33.

Do you have any other comments about the EME programme or any suggestions to the funders?

Survey questionnaire: unsuccessful applicants

About you

  1. What was your institution’s name at the time of EME grant application?
  2. What was your role at the time of EME grant application?

Your EME proposal

If you have had multiple unsuccessful proposals, please select one for the purpose of this survey where you have the most information available.

34.

What was the title of your EME proposal (optional)?

35.

Where did the original idea for the research topic come from? Select all that apply:

  • Academic researchers.
  • Clinical researchers.
  • Patient groups.
  • Health professionals.
  • Regulators.
  • Industry partner.
  • Literature.
  • Other.

Please provide further detail on your answer.

36.

Was the management and support provided by the EME programme team helpful for you during the proposal stage (e.g. the call guidelines; communication with the programme team during the proposal stage; support provided by the secretariat)? [Select one.]

  • Yes, the management and support provided by the EME programme team was very helpful.
  • Somewhat, the management and support provided by the EME programme team could have been more helpful.
  • No, the management and support provided by the EME programme team was not helpful.
  • I do not recall whether the management and support provided by the EME programme team was helpful.

Please provide further detail.

37.

Was the feedback provided by the EME Board helpful for you (e.g. knowledge and support provided by the EME Board and secretariat as part of the review process)? [Select one.]

  • Yes, the feedback provided by the EME Board was very helpful.
  • Somewhat, the feedback provided by the EME Board could have been more helpful.
  • No, the feedback provided by the EME Board was not helpful.
  • I do not recall whether the feedback provided by the EME Board was helpful.

Please provide further detail.

Your further research activity

38.

Did you continue working on the specific research idea/intervention after your proposal to the EME programme was unsuccessful? [Select one.]

  • I continued working on the research idea and conducted an efficacy trial.
  • I continued working on aspects of the research idea, such as mechanistic studies, but did not conduct an efficacy trial.
  • I did not pursue this research idea further, but I am aware that others have.
  • I did not pursue this research idea further and, as far as I know, it has not been tested elsewhere.
  • Other.

Please provide further detail on your answer.

39.

Did you apply for funding elsewhere for your specific research idea after your proposal to the EME programme was unsuccessful? [Select one.]

  • I submitted a grant application to another funding programme, and was successful.
  • I submitted a grant application to another funding programme, but was not successful.
  • I was funded by industry to continue working on the intervention.
  • I did not apply for funding elsewhere.

Please provide detail on your answer, including which funding programme you applied for, the level of funding secured and (if applicable) grant reference number.

Research progress and outcomes

40.

Has the specific research idea/intervention progressed from the stage it was at when you submitted the EME proposal? [Select one.]

  • Yes, the intervention has progressed, and was shown to be effective.
  • Yes, the intervention has progressed, and was shown to not be effective.
  • No, the intervention has not progressed.
41.

If yes, did the research lead to any outputs/outcomes? [Select all that apply.]

  • Yes, further research on the intervention led to a result that was published in a peer-reviewed journal.
  • Yes, further research on the intervention led to changes in policy and/or practice.
  • Yes, further research on the intervention led to commercial revenue (e.g. from spin-outs, licensing).
  • Yes, further research on the intervention has led to cost savings in the health system.
  • No, further research on the intervention has not led to any outputs/outcomes.

Please provide further detail on your answer, including web links where available.

42.

If no, would the research idea still be relevant to pursue? [Select one.]

  • Yes, the proposed research would still be relevant.
  • No, the proposed research would no longer be relevant as a result of recent research findings.
  • No, the proposed research would no longer be relevant as a result of recent changes in the guidelines and practice.

Please provide further detail on your answer.

43.

Please provide any other comments about the EME programme or any suggestions to the funder.

Interview questionnaire

Project background: aim, preparation, team

  1. Can you briefly describe the aim of the project, at its outset, and what you hoped to achieve?
    • What was the primary question you sought to answer? What was the health problem the trial sought to address?
    • How did you envisage the findings of the project to be progressed to the point where they are taken up into guidelines and/or NHS practice?
  2. Please outline the research that was to be conducted.
    • Did the project involve any mechanistic studies?
    • Did the trial involve any novel approaches? What was the trial methodology?
    • Did the project involve repurposing of an existing intervention?
  3. Could you describe how you prepared for the project?
    • What experience/insights did you base the proposed research on?
    • What was the origin of the research topic? Did the original idea come from academic researchers, clinical researchers, patient groups, health professionals, regulators, industry partner or literature – or a combination of these?
    • How did the project relate to your previous work (e.g. continuation of research programme, or ‘new’ to this research area)?
    • How was the preceding study funded, and by whom?
    • Did you identify any risks that could potentially undermine the success of the project? If yes, what were these and how did you mitigate them? (For example, did you conduct a feasibility study? Were there any considerations around background IP?)
    • Did you involve stakeholders [e.g. professional bodies, a registered Clinical Trials Unit (CTUs)] in the design phase of the project (i.e. before submitting the application)? If yes, please describe their role, how you interacted and how this shaped the project design.
  4. How was the project team organised?
    • Please describe the project team, and the roles and contributions of team members (e.g. skills, assets, infrastructure).
    • Had you worked with this team before? Please describe the level and nature of previous collaboration.
    • Did you collaborate with/work in partnership with industry, charities, (other) health-care professionals, international academic partners?
      • If yes, what specific role were these partners intended to bring to the project?
      • If no, why not?
    • Did the project receive any co-funding from other sources?
      • If yes, could you describe the co-funding arrangements (who were the co-funders, what were they contributing and at what level)?
      • If no industry co-funding, did you seek industry partnership funding? What was the reason (or reasons) this was not provided?
    • In hindsight, did the project draw on the right partners, or was there anything missing? Did the partners’ actual contributions match the original project plan?
    • Has the project led to further collaboration? Are you working with team members on other projects as a result of the EME-funded research?
  5. Were any external resources or infrastructure necessary for the project?

Trial experience and learning

44.

Was the project plan adjusted after the start of the project? If yes, how and why?

45.

Did you encounter any challenges to project implementation? If yes, what were they?

  • Were (some of) these unexpected? Could they have been anticipated?
  • How did you address them?
  • Could adjustments have been made to the project design to overcome these difficulties/what adjustments did you make?
46.

Did you have adequate resources or did you need to return for more funding?

47.

How were patients or the public involved in the delivery of research?

  • How many representatives were involved?
  • How often did you involve them?
  • Did patient representatives contribute to specific tasks?
  • How did patient involvement shape the project?
  • How did it affect the implementation of the project?
  • Did it enable the take-up of research findings into policy and practice?
  • How important do you think were these contributions?
48.

In hindsight, is there anything you would change about how the project was designed and implemented?

Key findings

49.

Could you summarise very briefly the key findings of the project?

  • Did the research provide a ‘conclusive’ answer to the research question?
  • Did the findings allow a decision on whether (or not) to take the research further?
  • What is the wider scientific and clinical significance of the findings?
  • If inconclusive, were there any aspects of the underlying research design that could have been altered to arrive at a conclusive answer (e.g. did the research involve the appropriate patient groups, participant numbers)?
  • If applicable, what were the findings of mechanistic work carried out as part of the project? What is the wider scientific and clinical significance of the findings?
50.

Has the project yielded any additional perhaps unexpected findings (including not anticipated at the outset of the project)?

Project outputs

51.

I have already read your publication on the main trial results, but were there any other publications directly stemming from the EME-funded research? Any other way you have disseminated the findings and learning?

52.

Tools, databases and sample collections, were any new research tools, databases or sample collections developed as part of the EME-funded project? Are these being used?

53.

Were there any patents, spin-outs or licensing agreements as a result of the EME-funded project?

  • If yes, could you provide further details?
54.

Have you used any advanced trial methodology?

  • If yes, what aspects of the trial were novel? What were the advantages and disadvantages of using this methodology?
  • Has this led to other investigators approaching you for advice?
55.

How has the EME-funded research contributed to capacity building?

  • At what levels (clinical research fellows, PhD, clinical investigator, etc.)?
  • What were the skills and knowledge improved as a result (e.g. disease/intervention specific but also related to translational research skills, regulation, industry standards, patient involvement, trial methodology, team working)?

Project outcomes

56.

Has the EME-funded work led to further research?

  • Has it facilitated further clinical research and/or development? By your group, or others?
  • Has it informed (or what is their potential to inform) further mechanistic/discovery research?
  • Has it enabled you to secure follow-on funding on the basis of the project findings? If yes, where from, what is the work funded, and at what level?
57.

If the EME project has led to further research, what have the follow-on studies achieved?

  • How does progress compare to the expectations you had at the outset of the EME-funded project?
58.

Have the project findings informed clinical applications, or do they have the potential to do so?

  • If yes, what are these? Did further development take place based on these project findings?
  • Who are current or future beneficiaries? Has the project increased the patient population who can access and benefit from this application?
  • If no, why not? What are the barriers?
59.

Have the project findings informed clinical practice guidelines and/or medical practice, or do they have the potential to do so? If yes, please explain further.

60.

Has the project contributed to any efficiency effects inside/outside the NHS (cost savings)? Have any health economic analyses/cost-effectiveness studies been conducted? If yes, could you provide details and references?

61.

Did the project lead to any commercial benefits (e.g. for the industry partners or background IP holders)? Has the potential market size for the developed application increased?

62.

Were there any other outcomes (including unintended)?

Funding landscape

63.

What have been the key changes over the past 10 years of conducting efficacy trials in the UK?

64.

At its inception, did the EME programme fill a gap in the funding landscape?

65.

Did it bridge the gap from proof of concept to HTA/effectiveness studies and commercialisation/implementation?

66.

Has the EME programme’s role changed over time? To what extent does it complement other funding streams today?

67.

What funding programme could you have applied to if the EME scheme had not been available? Are there alternative sources of funding for this type of research? How do alternative funding sources differ from the EME scheme?

68.

Are there synergies or duplications with other funding programmes?

69.

What gaps remain in certain research areas/disciplines or specific types of efficacy research?

70.

How was the research underpinning the EME-funded project we discussed earlier funded?

71.

Where have you/would you look for funding to further develop a promising intervention following on from an EME grant?

Design and management of the EME programme

72.

Is the EME programme an appropriate funding mechanism, in terms of funding amount, project length and scope?

73.

Are there aspects of the scheme’s current design and requirements that you consider a barrier to achieving its aims?

74.

Are there additional activities the EME programme could have supported that you think would have made the research more effective and increased the potential for impact?

75.

Was the management and support provided by the EME programme staff and review committee helpful for you?

Final comments and close

76.

Do you have any other comments about the EME programme or any suggestions to the funder?

Copyright © 2021 Rentel et al. This work was produced by Rentel et al. under the terms of a commissioning contract issued by the Secretary of State for Health and Social Care. This is an Open Access publication distributed under the terms of the Creative Commons Attribution CC BY 4.0 licence, which permits unrestricted use, distribution, reproduction and adaption in any medium and for any purpose provided that it is properly attributed. See: https://creativecommons.org/licenses/by/4.0/. For attribution the title, original author(s), the publication source – NIHR Journals Library, and the DOI of the publication must be cited.
Bookshelf ID: NBK575345

Views

  • PubReader
  • Print View
  • Cite this Page
  • PDF version of this title (1.9M)

Other titles in this collection

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...