Chapter 2A scoping review and stakeholder consultation charting the current landscape of the evidence on online patient feedback

Summary

As the initial project, this scoping review and stakeholder consultation aimed to identify and synthesise the current practice, state-of-the-art practice and future challenges in the field of online patient feedback. We searched electronic bibliographic databases and conducted hand-searches up to January 2018. We included primary studies of internet-based reviews and other online feedback (e.g. from social media and blogs), from patients, carers or the public, about health-care providers (individuals, services or organisations). Key findings were extracted and tabulated for further synthesis, guided by the themes arising from a consultation with 15 stakeholders with online feedback expertise from a range of backgrounds, including health-care policy, practice and research. We found that, as with much digital innovation, research is lagging behind practice. The current literature helped to clarify the frequency of online commentary and challenged the assumption that feedback is usually negative. The review identified gaps in the evidence base, which can guide future work, especially in understanding how organisations can use feedback to deliver health-care improvement.

Method

When we began this work, to the best of our knowledge, no synthesis of the existing body of literature on online patient feedback (reviews and/or ratings) had been conducted. Collating knowledge and developing an understanding of current research was an important precursor to further work in this area. Adopting a scoping review methodology allowed us to access and review existing evidence, summarise and disseminate research findings and identify gaps in the existing literature.

Scoping studies ‘aim to map rapidly the key concepts underpinning a research area and the main sources and types of evidence available’.³² They are useful when reviewing literature on complex topics or areas that have not been reviewed before. The depth of the subsequent analysis of findings depends on the purpose of the review.³³ Unlike other types of reviews, such as quantitative systematic reviews, the scoping review does not appraise the quality of research evidence. However, it does consider the strengths and limitations of individual studies and critique the existing body of knowledge.

To identify relevant literature, a list of free-text and thesaurus terms likely to retrieve articles about online patient feedback was compiled using an iterative process of consultation between the research team and an information specialist.

Searches were run in May 2015 and updated in January 2018. Five databases were searched: MEDLINE (In-Process & Other Non-Indexed Citations and Ovid MEDLINE, 1948–present, accessed through OvidSP), EMBASE (1974–present, accessed through OvidSP), PsycINFO (1967–present, accessed through OvidSP), Cumulative Index to Nursing and Allied Health Literature (CINAHL) (1981–present, accessed through EBSCOhost) and Social Science Citation Index (1956–present, accessed through Web of Knowledge). Titles and abstracts were subsequently screened for relevance using the following inclusion criteria.

• Topic area: the main focus of the article had to be about online feedback (to include internet-based reviews, ratings and other online feedback, such as found in social media and blogs) from patients, carers and/or the public, about health-care providers (individuals or organisations).
• Type of paper: original research.
• Study design: all study designs.
• Date: 2000 to present.

Titles and abstracts were screened independently by two authors (AMB and VW) using Covidence (Veritas Health Innovation, Melbourne, VIC, Australia), a software package designed to aid the screening process. Disagreements were resolved in discussion with a third author (JP). Full texts were screened using the same criteria and process (again by two authors, with referral to a third in cases of disagreement). All included articles can be found in Report Supplementary Material 1. We have used a Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram to illustrate this, although as this was a scoping review of diverse studies we do not follow PRISMA reporting more generally (Figure 1).

FIGURE 1

Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram demonstrating the final search and screening process.

Once full-text articles had been selected, they were randomly assigned to one of two authors for single data extraction. Data were extracted and tabulated using a standard pro forma (in a scoping review, it is not necessary to conduct double extraction). The following data were extracted: information on authors, date of publication, study aims, sample, methods and findings. Reference lists were checked for further articles to include.

The articles were thematically grouped by aim for further analysis in a process of data charting. A descriptive narrative synthesis of the findings was produced through individual analysis and discussion within the research team.

Stakeholder consultation

We conducted a stakeholder consultation with 15 representatives from a range of organisations, including policy-makers, senior clinicians working in patient experience, patient experience managers, regulators, representatives from patient feedback organisations and service users who have read or provided feedback. They had all collected or used online feedback. The aim of this exercise was to identify stakeholder priorities and questions to guide the literature review. In other words, we wanted to respond to the preoccupations of stakeholders and to see the extent to which the current evidence base could address the questions they had about online feedback. They were consulted about their perceptions and concerns about online feedback, including what they thought was important for the future. Consultations were conducted individually, in person or on the telephone, and notes were taken to capture the data, which were then analysed inductively. Ethics approval was not required.

In the stakeholder consultation, we identified six key issues to help navigate the online patient feedback landscape, which addressed evidence gaps identified by stakeholders:

1. Who provides and who uses reviews?
2. How do organisations currently use reviews?
3. What is the content of reviews?
4. Why is online feedback given?
5. What are staff and service user attitudes towards online feedback?
6. How reliable is online feedback?

Findings

Where, when and what kind of research has been conducted?

The majority of the 78 included papers described studies conducted in the USA (n = 44^{15,26,29,34–74}). Others were from the UK (n = 12^27,75–85), Germany (n = 8^{11,28,86–91}), the Netherlands (n = 3^49,92,93), China (n = 3^94–96), Austria (n = 1⁹⁷), Canada (n = 1⁹⁸) and Switzerland (n = 1⁹⁹). Five studies^{10,21,100–102} were conducted using patient feedback collected in more than one country.

As presented in Table 1, the majority of included studies used quantitative methods and were predominantly exploratory or descriptive cross-sectional studies, surveys or experiments, or employed machine learning. There were also qualitative and mixed-methods studies.

TABLE 1

Scoping review: all included studies presented by design

Who provides and who uses online reviews?

From the literature, it is apparent that public awareness of rating sites differs, that at present the numbers of people providing online reviews are still low, but people are starting to use these sites more frequently. A German survey²⁸ showed that 32% of the public were aware of health rating sites (people were more commonly aware of rating sites for other products and services) and health rating sites were seen as less important sources of health information than other sources (e.g. recommendations of friends and family).⁵⁴

Posting (providing) a rating was a slightly more established activity in Germany than in other countries: a German survey⁸⁷ (2013) showed that 11% of participants had posted online feedback. In studies in Austria (2014)⁵⁴ and in the USA (2015),⁹⁷ the prevalence was 6%.^54,97 The most recent UK figure, as identified in our own Improving NHS Quality Using Internet Ratings and Experiences (INQUIRE) survey (see Chapter 3), was 8%.⁷⁷ Other studies showed that women were more likely than men to post feedback.^89,90

People who provide online feedback are likely to be younger, have higher levels of education⁸⁹ and have a long-term condition.^52,89 Likelihood to use online review sites may also be influenced by the doctor–patient relationship:⁷⁵ perceiving the relationship to be friendly, feeling listened to and being the same sex as the general practitioner (GP) seem to predict use, and willingness to use was predicted by autonomy in health-care decisions. However, patients who felt that they had clear explanations from their GP were less likely to use online review sites. Men and those with less formal education were less likely to use these sites,^52,89 as were people with higher incomes.⁷⁵

How are online reviews used by organisations?

From this scoping review, it was evident that this is a clear gap in the literature: no papers that considered the purpose of online patient feedback or uncovered the practices and processes governing its use in health-care organisations were found. However, evidence that some services have begun to incorporate online reviews into service improvement was uncovered: in Germany, a survey⁸⁸ found that ophthalmology and gynaecology services were the most likely to implement change based on online patient feedback. Similarly, there was limited research on the value of online review in health-care inspection or monitoring agencies, although its potential was noted despite some concerns, for example by staff in a study of the Dutch Health Inspectorate.⁴⁹ This is particularly true of structured patient feedback websites, which were thought to contain more pertinent additional information than other social media platforms [e.g. Facebook (Facebook, Inc., Menlo Park, CA, USA; www.facebook.com) and Twitter (Twitter, Inc., San Francisco, CA, USA; www.twitter.com)].⁹³ The structured websites were considered by patients to provide ‘on the ground’ or ‘bottom-up’ quality monitoring.

What is the content of reviews?

Characteristics of reviews

Strikingly, the included studies repeatedly showed that the majority of reviews were positive and that numeric ratings for health-care providers tended to be high.^51,90,91,100 Reviewers often recommended the health service to other patients.^85,100 Positive reviews were more likely to be posted by females, older adults and those with private health insurance.⁹¹ Having a long-standing relationship with a health-care professional was also linked to providing a positive review.⁶⁹

Reviews tend to be short. A sentiment analysis of 33,654 reviews of 12,898 medical practitioners in the New York State area found that, on average, reviews were 4.17 sentences long and 15.5% contained only one line of text.⁵⁸ Lengthier commentaries were more likely to be negative.⁹⁰ When family members reviewed health services, they were more likely to comment on matters of patient safety.⁶⁸

In general, comments tended to concern services or providers, clinical and administrative staff, and the physical environment. They often related specifically to clinicians and focused on knowledge and competency,^10,98 patient-centred communication,^10,95,98 personal character traits,^10,29 professional conduct,¹⁰ dignified care¹⁰⁰ and co-ordination of care.⁶⁹ Waiting times and length of appointments often featured in the reviews^29,45,95,100 and other themes focusing on the service or environment pertained to cleanliness,^67,100 scheduling appointments,⁶⁷ insurance,⁴⁵ access,⁶⁹ administrative staff^45,69 and parking.¹¹ Again, these facets of patients’ experiences were more frequently commented on in positive (rather than negative) terms.⁷⁸

Who is reviewed?

Male staff were more likely to be the subject of reviews than female staff, who were more likely to receive positive feedback.^87,91 When comments and ratings were specifically aimed at professional staff, they included mainly generalists.²⁹ Two studies indicated that some specialties (e.g. radiology) were less frequently commented on and some subspecialties received a higher number of reviews (e.g. facial plastic surgery) than other services.⁵¹ Two studies showed that surgeons were reviewed quite frequently on German and US websites and plastic surgeons in California had a large number of online ratings and reviews.⁷³

How and why do service users use these sites?

In general, the included studies report that these sites are used to post feedback, or to help choose a doctor or another health professional (e.g. dentist).^{28,42,52,54,97} Twenty-eight per cent of respondents in a 2014 US survey⁵⁴ had used reviews and ratings websites to find a doctor. A 2013 German survey²⁸ showed that 25% of respondents had used the websites for this purpose. In the latter survey, 65.35% of the 1505 respondents had chosen a doctor based on the reviews and ratings, whereas 52.23% had used the online feedback to identify which doctors to avoid. In a nationally representative survey in the USA,⁴² 35% of those who had used rating sites in the last year said that good ratings had a positive effect and 37% said that poor ratings had a negative effect on physician choice. Further evidence of the impact of the valence of reviews on physician choice was found in an experimental study, which confirmed that negative review content reduces the willingness to choose a doctor and that presenting negative reviews before positive ones has a greater (negative) effect than if the positive reviews are presented first.⁵⁶

Qualitative research exploring the motivations to post or read online reviews is limited. An English interview study⁸² with primary care patients who had never posted feedback found that they suggested they would do so only if they wanted to review an extremely positive or extremely negative experience at their general practice. They did not see the value in providing feedback on routine or ordinary experiences.

The number of negative reviews read, and the order in which they were read, was also found to have an impact. Reading negative reviews before positive reviews led to patients becoming less willing to consult a particular doctor.⁵⁶ Characteristics of the reviewers, including perceived trustworthiness, credibility and expertise, were also found to have an impact. When it came to content, fact-oriented reviews were reported as preferred over emotional-oriented reviews, or those containing slang or humour.⁹⁷ In addition, those who used a rating or review website to read comments were then more likely to rate a health-care experience in future.⁸⁹ Patients are more likely to spend more time on websites that contain comments (i.e. not just numeric ratings),⁵⁵ which, the authors speculate, may increase the potential for ‘suboptimal choices’.⁵⁵

How do staff and service users feel about online feedback?

Based on the extant literature, health professionals hold a range of concerns about online feedback, but patients’ attitudes are more varied. Further to the research reported earlier about the perceptions of monitoring agency staff, three studies using interviews and surveys explored health professionals’ views.^43,70,81 In a qualitative interview study in England, GPs expressed their apprehension, particularly on the validity and representativeness of online feedback.⁸¹ Twelve per cent of doctors responding to a survey deemed online rating websites useful and 39% agreed with the feedback they had received.⁴³ In another survey, 65% of US hand surgeons said that they were sceptical of feedback websites, with 82% reporting that it had no implications for their practice.⁷⁰

Patients have varied attitudes towards online feedback. A qualitative interview study⁸² about reviews and ratings in general practice in England found that participants questioned the need for online feedback and were unsure if GPs would use it. For some, the benefits of online feedback were that it could be posted remotely, could be shared with other patients and would be taken seriously by GPs. Others were concerned about privacy and security, and believed that online feedback could be ignored. In a qualitative study⁹⁹ conducted in Switzerland, parents reported that review websites were more like a directory of services than a decision aid, asserting that there was not enough information to guide a choice. For them, the most effective way to evaluate a health professional was to do so in person.

How reliable are online ratings and reviews?

Comparisons have been made between traditional measures of experience and satisfaction, such as the NHS Inpatient Survey in England and the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) in the USA, and feedback on rating and review websites (e.g. Yelp or NHS Choices). These have shown similarities between online feedback and standardised measures of patient satisfaction and experience, although the online reviews tend to contain more information.

Online reviews and ratings of specific hospitals were correlated with survey responses in both England and the USA.^34,58,76,79 A strong correlation (r = 0.49; p < 0.001) was found in the USA between Yelp scores and overall scores on the HCAHPS.³⁴ In an English study,⁷⁶ the number of patients willing to ‘recommend the hospital to a friend’ was correlated with a hospital’s overall rating on the national inpatient survey (Spearman’s ρ = 0.41; p < 0.001). Weak correlations were established between positive online recommendations and lower hospital mortality ratios (Spearman’s ρ = –0.20; p = 0.01), and better ratings of hospital cleanliness were weakly associated with lower rates of infections, particularly meticillin-resistant Staphylococcus aureus (MRSA) (Spearman’s ρ = –0.30; p = 0.001) and Clostridium difficile (C. difficile) (Spearman’s ρ = –0.16; p = 0.04).⁸⁵

Analysis of online reviews in both England and the USA showed that their content was similar to the domains on patient surveys, suggesting that the surveys do cover items that matter to patients. Significant associations were found between scores on the NHS Inpatient Survey in England and online feedback on domains such as hospital cleanliness, dignified and respectful treatment, and involvement in decisions about care.⁷⁹ However, studies in the USA showed that more topics were raised in online feedback than on the HCAHPS,^34,62 indicating the potential for it to supplement existing measures of patient experience. However, the additional topics may have been more salient to some services than others.⁵⁸

Discussion

As with many digital health innovations, the research of the field of online ratings and reviews lags behind the practice and the issues of interest to stakeholders. We know that there are many websites collecting online patient feedback and we know that people are using them; however, this scoping literature review has shown that the current evidence base is limited to a relatively small number of, often small-scale, studies from which it is hard to draw definitive conclusions. Our initial consultation with stakeholders about their priorities helped guide our questions, but it is clear from the literature that, as yet, current research does not address all areas of interest.

We can conclude that patients in several high-income countries are using online feedback sites to choose health professionals and to gauge public opinion about them, and that this use is increasing. These sites are also beginning to be used to monitor health services, especially in the Netherlands, demonstrating their potential for the care quality regulators.

Papers examining the content of online reviews reveal that patients commented on a range of factors about their health-care experience, including waiting times, environmental factors and staff.¹⁰⁰ Comments about staff predominantly related to medics themselves and centred on their perceived knowledge, skills, competence and communication ability.^10,42 Such findings illustrate that patients can, and do, comment on a range of aspects of their experience. Studies also consistently demonstrate that the majority of reviews are positive, and that negative reviews may be influential both in terms of their content¹⁰⁶ and the order in which they are read.⁵⁶ Several studies found that negative reviews tend to be expressed with more words than positive comments.^63,90 This could mean that negative comments provide detail that could be used in locating and addressing the problem. Such findings also have implications for the design of online platforms for capturing feedback, which should allow the option of free-text comments, as well as check boxes and scales.

In general, online feedback has been shown to complement standardised patient surveys and can correlate with other measures of quality. Two US studies (Bardach and colleagues⁶⁸ and Ranard and colleagues⁶²) showed that more topics were raised online than in a patient survey. We can speculate that without the constraints of a structured survey, patients might be able to provide a more diverse range of data for use in quality and service improvement.

Few studies have focused on the attitudes and perceptions of health professionals in relation to online patient feedback. Patel and colleagues⁸¹ found that health professionals were concerned about its usability, validity and transparency. Our initial search uncovered numerous editorials and opinion pieces written by, and for, health professionals who were sceptical about online reviews.

Two other literature reviews have sought to examine the research in this field. Verhoef and colleagues¹⁰⁷ followed Arksey and O’Malley’s³³ protocol to conduct a scoping review of literature about the relationship between quality of care and social media and rating sites. Their 29 papers included opinion pieces and original research, and focused on the relationship between social media and care quality.

Emmert and colleagues‘²⁸ systematic review aimed to answer eight questions about the percentage of physicians who were rated, the average number of ratings, the relationship of rating with physician characteristics, whether ratings were more likely to be positive or negative, the significance of patient narratives and the problems with rating sites and how they could be improved. The current review provides updated information on these questions; we have provided a synthesis of research on the content of patient comments, a more complete description of users of these sites, including patients who post reviews or are influenced by the reviews they read, and other use by inspectorate bodies.

Conclusion

By systematically searching for and presenting research evidence that addresses the preoccupations of the stakeholders we consulted, this scoping review charts the current landscape of online patient feedback research.

We have demonstrated that research in this area has emerged rapidly in recent years, but remains limited in both quantity and quality, given the spread of the phenomenon of online feedback. Many of the concerns of stakeholders remain unaddressed in the extant literature and therefore informed our own primary work in the INQUIRE project. For example, in the next few chapters we describe our findings about which patients provide and use reviews, and why do they do this; what the attitudes of professionals are towards online feedback; and how health-care organisations approach online feedback. The evidence gaps also inform our other recommendations for further research presented in the final chapter (see Chapter 8).