Challenges in Stakeholder Involvement

We implemented a Web-based discussion board in preference over a series of teleconferences because of what we believed would be advantages of the online approach. These included: greater flexibility for stakeholders to participate in the discussion at times convenient to them, including nonworking hours; a platform that would allow everyone to have a more equal voice, where discussion would not be led by the stakeholder who verbally dominated; full participation by all stakeholders in all discussions, not just those that occurred during calls they were able to attend; a full record of all discussions, without the need to summarize verbal discussions, which inevitably leads to omissions and other errors; less time expenditure by stakeholders who would not be asked to sit through numerous 60- to 90-minute phone calls; considerably less resource expenditure by Evidence-based Practice Center (EPC) staff, where each teleconference hour translates into 7 person-hours of resources; and less opportunity for “multitasking” during discussion (e.g., answering emails while participating in teleconferences).

However, we faced several challenges when implementing the SharePoint Web site discussion board. Perhaps the greatest challenge was the low participation rate of stakeholders during most stages of the project. When the discussion period was announced, few participants logged onto the Web site within the first week. Because of the low log-in rate, combined with the low number of page views for most stakeholders, the discussion period had to be extended. Despite the extension of the discussion period, low participation rates persisted. To increase participation, we sent several email reminders to stakeholders. After the discussion period was closed, participation rates remained low during the nomination period, despite simplification of the process by asking for nomination by email. We subsequently telephoned remaining stakeholders, which increased total participation. Overall, although SharePoint offered the convenience of asynchronous collaboration it seemed to lack appropriate incentives to engage stakeholders. Notably, the seemingly egalitarian Web-based workspace failed to connect patient advocates with the other stakeholder representatives. Assembling patient advocates independently via group teleconference proved more effective, but limited the integration of patient perspective into the SharePoint discussion forums.

Apparent barriers to using the Sharepoint Web site more fully included technical trouble logging in (e.g., while traveling); apparent reluctance by stakeholders to devote unscheduled time to sign in to the Web site, review the discussions and comment; hesitancy commenting in the online discussion by stakeholders without a technical background; and possibly a lack of engagement and interaction from communicating by text instead of verbally. The large number of topics may have been too great a burden for stakeholders to read at a single sitting. In addition, the timing of the project during the summer months may have limited participation because of vacation schedules. A large part of the problem of poor participation may have been simply that the Web-based approach did not force people to schedule a time to participate (as they would for a teleconference), thus there may have been a lack of perceived urgency. Furthermore, for almost all topics, the comments were so sparse that there was little sense of a discussion for stakeholders to participate in.

Several possible solutions exist for these problems. One potential approach that might improve discussion on an online discussion board would be to increase the depth of topic discussion by reducing the number of stakeholders. This could be accomplished by purposely limiting the number of participants to those who show the ability and proven willingness to participate in discussions at early stages. For example, a run-in phase could allow inclusion of stakeholders, who could be given the opportunity to offer new topics and start discussions, but then further discussion would continue with only the interested stakeholders. In effect, this is what occurred, but an a priori plan to limit the number of participating stakeholders could help to make the process more explicit and possibly shorten the time period required for discussion. A potential downside of this approach is that it could be seen as more explicitly biased since certain stakeholders would be dropped from further participation. Also, the practical problem of how to cordially disinvite stakeholders from further participation would have to be overcome.

Alternatively, the EPC could revert to the more standard and basic approach of conducting a series of teleconferences, where all available stakeholders could join any or all calls. The teleconferences could have either a loose agenda where any new or old topic could be addressed, or a more structured agenda where each call would focus on a general category of Future Research Needs (FRN) topics. We used this approach near the end of the discussion period, when we noted that the consumer stakeholders (the patient, the patient advocate, and the representative from the transportation industry) had not participated in the discussion. We organized a separate teleconference for the three of them and a 1.5 hour teleconference produced nine new topics. The EPC would need to work to ensure that all participating stakeholders are given multiple opportunities to fully express their thoughts. The EPC would also have to accurately summarize the calls in an unbiased fashion, a nontrivial task. The summaries would have to be in a form that could be relatively easily used by the stakeholders.

A similar approach could be to use focus groups to allow a full, simultaneous discussions of each topic. Focus groups could be formed by either collecting stakeholders with similar backgrounds together to focus on specific topics or by randomizing stakeholders to focus groups in order to obtain a balance of viewpoints. These approaches could allow more in-depth discussion of focused topics. However, potential downsides include lessening the participation of less vocal or less topic-knowledgeable individuals and setting up separate subgroup discussions that not all stakeholders have equal access to. This approach may be somewhat simpler to organize and would require less time commitment by stakeholders than multiple teleconferences for all stakeholders.

An approach that combined both teleconferences with online discussions may be able to combine the advantages of the two discussion types, without many of the disadvantages of each. However, this approach is likely to be more time-consuming than the original timeline planned (though possibly less time-consuming than the actual discussion period for the current project). One variation could be an initial gathering of FRN topics and rationale text online, followed by teleconferences to further discuss each topic. The teleconferences would be summarized by the EPC staff and uploaded to the Web site. Further discussion would then be encouraged on line. Clearly, numerous other similar approaches are possible.

Regardless of which type of discussion were held, there may be advantages to limiting the size of the stakeholder panel that could improve discussion. We took the approach of being expansive to include as many voices as feasible. However, in future, it may be better to limit the panel to a single member of each stakeholder category. Attempts should be made to choose stakeholders who show enthusiasm in joining the panel and participating in discussions. However, unfortunately, across all the stakeholders we did not find a strong correlation between the level of enthusiasm upon invitation and the degree of participation in discussions.

In summary, the online Web site discussion of numerous topics by a large stakeholder panel was only moderately successful. Various approaches to improve participation and discussion are possible, including increased use of teleconferences, restricting the size and members of the stakeholder panel, and other approaches. More experience with different approaches is needed.