An official website of the United States government
NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.
Battles J, Azam I, Reback K, et al., editors. Advances in Patient Safety and Medical Liability [Internet]. Rockville (MD): Agency for Healthcare Research and Quality (US); 2017 Aug.
Shawn L. Mincer, MSW, Michael J. Lee, MD, Richard J. Bransford, MD, Saint Adeogba, MD, Karen L. Posner, PhD, Lynne S. Robins, PhD, Pornsak Chandanabhumma, MPH, Michelle S. Lam, BS, Aaron S. Azose, BS, and Karen B. Domino, MD, MPH.
Author Information and AffiliationsEffective communication is a key ingredient of safe, high-quality health care. Shared decision-making (SDM) is a collaborative decision-making (DM) process between the patient and provider for preference-sensitive treatment decisions. The goal of SDM is to empower patients to participate as active partners in their health care decisions. The Shared Decision-Making in Surgery to Improve Patient Safety and Reduce Liability investigators implemented SDM in two orthopedic spine surgery clinics. The purpose of this paper is to identify barriers and solutions for shared decision-making that resulted from this study.
Patient activation brochures and decision aids (DAs) were developed and disseminated. Pre-implementation patient surveys assessed baseline SDM elements in routine practice. SDM education and training materials were developed, and surgeons were trained. Observation assessments were used with surgeons during training. Clinical DM interactions were audiotaped before and after training and assessed for the presence or absence of SDM elements.
Barriers to SDM implementation included adoption and dissemination of patient activation materials and DAs, concern for increased time commitments, surgeons’ beliefs they were already doing SDM, and entrenched clinic procedures. An SDM “Train-the-Trainer” Toolkit was developed to overcome training barriers and disseminate the methodology. Trainer observation and surgeon self-assessments of clinical encounters suggested that the SDM elements most overlooked were a patient seeking input from trusted others (65 percent), establishing a patient’s role in DM (53 percent), teach-back (42 percent), eliciting patient preference (24 percent), and communicating uncertainty (24 percent). Pre- and post-training audiotaping of clinical DM showed surgeons made minimal improvement in the element establishing the patient role in DM. No surgeons used teach-back to check for understanding prior to SDM training, and only one surgeon incorporated teach-back into his clinical interactions after training.
In conclusion, physicians in this study did not include all elements of SDM during clinical DM with patients. Successful implementation of SDM requires a process to engage stakeholders, identify barriers, and develop interventions to overcome barriers, including novel teaching strategies and data-driven individual feedback.
Effective communication is a key ingredient of safe high-quality health care. Shared decision- making (SDM) is a collaborative decision-making (DM) process between the patient and provider for preference-sensitive treatment decisions that takes into account patient preferences and values.1,2 The goal of SDM is to empower patients to participate as active partners in their health care decisions.3 The Shared Decision-Making in Surgery to Improve Patient Safety and Reduce Liability investigators implemented SDM in two orthopedic spine surgery clinics to better explore barriers and opportunities for shared decision-making as a means to improving patient safety.
Poor communication between health care providers and patients represents a risk to patient safety. Informed consent with patient “teach-back” of key information about the proposed treatments or procedures is part of the 2009 National Quality Forum’s (NQF) Safe Practices for Better Healthcare.4 The NQF’s Consensus Panel had great concern with the frequency with which patients do not receive adequate informed consent. As communication failures between patients and health care providers are at the root of systems’ failures and human errors that lead to harm,5 the NQF Consensus Panel agreed that communication is key to preventing patient harm related to lack of informed consent. Better-informed patients serve as an additional layer of protection against medical errors.6
A Washington State statute provides specific protections from lawsuits if SDM occurs.7,8 As part of such SDM, decision aids (DA) that are peer-reviewed and use patient-friendly language must be used. Also the SDM process must include “teach-back” from patient to provider. The intent is to produce an improved informed consent process that is both standardized by procedure and more informative for patients. It contains elements to ascertain that the patient understands and appreciates the treatment options and their risks and benefits.
SDM recognizes that patients and health care providers each bring their own unique perspectives, preferences, and values to the DM process.3 SDM attempts to integrate these sometimes different perspectives into a cohesive process of DM. This process recognizes that health can be defined as “a dynamic state of well-being characterized by a physical and mental potential, which satisfies the demands of life commensurate with age, culture, and personal responsibility.” 9 Our study used an SDM model composed of nine core elements,10 plus the addition of the “Teach-Back” component required by Washington State statute8 (Table 1).
SDM Elements and the Patient Teach-Back Component
Patient activation is a key early component of the SDM process. Since SDM is not the current norm in health care, the concept and expectation for SDM needs to be introduced to patients prior to the clinical encounter so they will be prepared to engage in SDM with their physicians. Patient activation leads to better health outcomes and care experiences.11,12
We implemented SDM for patient preference-sensitive orthopedic spine treatment in orthopedic spine surgery clinics in two hospitals within the university health care system. Our primary aim was to introduce SDM to the hospitals through these two clinical settings, identify barriers in achieving this aim, and develop strategies to mitigate the barriers. In order to accomplish this aim, we developed and disseminated patient activation and DAs to patients, developed a provider training module on conducting SDM during the clinical encounter, and assessed providers’ success in following SDM elements in their treatment discussions with patients.
After approval by our institutional review board (IRB), an implementation process occurred in two university hospital orthopedic spine clinics. Implementation of SDM required a multi-stage process engaging all levels of the organization, including institutional approval, stakeholder engagement, development/identification of patient activation and DA materials, provider training, dissemination of patient activation and DA materials to patients, and ongoing assessment and provider feedback.
We conducted a series of meetings with senior hospital administration to obtain approval for adoption of SDM into the current informed consent regimen and to facilitate implementation. The decision was made to conduct a limited pilot of SDM before deploying this new process more broadly within the health care system.
Project leaders met with all stakeholders throughout the implementation process. There were two core stakeholder groups in each of the two orthopedic spine clinic locations: physicians (including residents, if applicable) and physician assistants (hereafter referred to as “surgeons”), and medical staff and clinic administration (hereafter referred to as “staff”). Surgeons were the primary personnel implementing SDM with patients during the clinical encounter. Ongoing engagement and support from surgeons were a crucial component of SDM implementation.
A patient activation brochure was created by the project team and edited by the Patient Education Services in the health care system to assure patient-friendly language and consistency with health system document style. The brochure introduced the concept of SDM and provided tools that patients could use to be more engaged in the DA process with their provider. The activation brochure was intended to function as a catalyst to foster patient engagement in the SDM process. Activation brochures were designed to be disseminated to a broad range of patients, regardless of presenting condition or reason of visit, prior to the visit. Patient activation brochures were given to staff for inclusion in patient hand-outs and mailings.
Meetings were held with the orthopedic spine surgeon group to develop procedure-specific DAs for dissemination to patients. We gathered together all the health care system’s orthopedic patient education materials and developed DAs that were consistent with them but were written in patient-friendly language, presented evidence-based risks/benefits, and included required elements of SDM. Due to surgeon disagreement with the specific content and quality of the evidence base of the DAs, the surgeons agreed to use two commercially available DAs that were purchased from Health Dialog, Inc. (Boston, MA) for two conditions: herniated disc and spinal stenosis.13,14 DAs were given to surgeons, physician assistants, and patient care coordinators for dissemination. Desk copies were disseminated to all clinic staff.
Following administration and surgical group approval, meetings were conducted with clinic managers and other staff to obtain their support for the implementation process. Clinic managers played key roles in the implementation process. They identified members of the clinic team that were best suited to carry out implementation tasks, assisting in coordinating ongoing project support tasks as needed. These managers were in control of the actual facility and identified where materials could be posted or placed in the clinic space, such as copies of training materials and DAs. Patient care coordinators were a key component as they had ongoing communication with a patient prior to, during, and after a visit. Strategies and tools for SDM implementation were developed with active participation by staff.
Prior to implementation, an anonymous patient exit survey was conducted in the two orthopedic spine clinics to assess surgeon performance of specific elements of informed consent and SDM. Patients who were visiting for a new problem or a pre-operative visit were eligible to participate in the survey. Performance elements were: nature of the condition or procedure, alternatives or choices for treatment, risks and benefits, and preferences. For each item, we included a follow-up question: Did he/she use words you could understand? We also elicited general satisfaction with the clinic visit and provided an opportunity for open-ended responses. All surveys included demographic information such as age, sex, education, and overall health status that have been found to be correlated with patient satisfaction. The survey also elicited information about how patients prepared for their visit (eliciting input from trusted others, searching the internet, consulting written resources, and writing down questions to ask).
Surgeon training involved a short, 20-30 minute, individual or small group training on SDM. Participants were provided with pocket reminder cards, and posters were placed in the workplace that had reminders and prompts on how to conduct SDM. They were exposed to the patient activation brochures and DAs. This was followed with “on-the-fly coaching” during clinical encounters. Providers were observed, and an SDM “scorecard” was kept by the observer. Providers completed matching self-assessments. These were compared, and “on-the-fly” coaching was provided on missing elements. These trainings occurred over a 3-week training period and were done in the clinic to decrease additional time burdens on already busy physicians.
Clinical DM encounters between surgeons and patients were recorded before and after implementation of SDM. All patients and surgeons subject to these procedures provided written informed consent. New patients and patients making pre-operative visits were eligible to participate. Visits in which treatment options were not discussed (e.g., referral to a different specialist, diagnostic discussion with orders for further testing to determine the nature of the condition) were excluded.
All recordings were transcribed for scoring. Two investigators/staff members scored a sample of transcriptions to ascertain inter-rater reliability. Once inter-rater reliability was determined, the remaining transcripts were scored by the two same raters, and consensus was used to establish a reliable final score for the surgeon. Scoring was conducted using the IDM-18 scoring system of Braddock,10 giving 0-2 points for each of the nine elements of SDM evident. In addition, a 10th element of “teach-back” was scored to address the specific requirements in Washington State for SDM.8 Scores from the pre-implementation phase were compared to post- implementation to assess whether providers improved their performance of SDM after the training. We compared scores before and after training by linear mixed model on 61 total transcripts to assess training effect. We compared the presence of individual SDM elements before and after training with Fisher’s exact test with p<0.05 considered statistically significant.
The results from this study describe a patient survey conducted before implementation of SDM. Results were also obtained through trainer observations and surgeon self-assessments of their clinical encounters during training and also by scoring transcripts of surgeon-patient clinical interactions before and after the intervention. We also summarize several implementation barriers and solutions for both staff and surgeons. These barriers and their solutions are described in Table 2, and we will elaborate on findings for each group, separately.
SDM Implementation Barriers and Solutions by Surgeons, Medical Staff, and Research Team
In pre-implementation planning meetings, surgeons expressed the belief that they were already conducting SDM with patients. Our pre-implementation survey of 137 patients demonstrated that this was not the case. Although patients reported that surgeons explained their condition(s) in 93 percent of the cases, the surgeons failed to discuss many elements of SDM. They failed to tell patients that there was more than one treatment choice 23 percent of the time, failed to discuss the pros and cons 25 percent of the time, and failed to elicit patient preference 26 percent of the time during clinical encounters. The survey results showed that surgeons were effective in discussing the clinical nature of the visit, but they were less effective in engaging the patient as a partner in the DM process.
Similar to the pre-implementation survey, results from training assessments showed that providers were most effective in discussing the nature of the clinical condition and less effective in engaging patients as partners in DM. Trainer observation and surgeon self-assessments of clinical DM during implementation suggested that the SDM elements most overlooked were seeking input from others (65 percent), establishing the patient role in DM (53 percent), teach- back (42 percent), eliciting patient preference and communicating uncertainty (24 percent each), and alternatives (18 percent) (Figure 1).
Elements of Shared Decision-Making Included in Surgeon-Patient Decision-making Assessed by Trained Observers
There was improvement in the element establishing the patient role in DM (17 percent before vs. 41 percent after training, p=0.049, Figure 2). No surgeons used teach-back to check for understanding prior to SDM training; one surgeon incorporated teach-back into his clinical interactions after training (0 percent vs. 31 percent, p=0.131). A mild, but non-significant, improvement in overall scores of approximately 1 point was observed (effect [regression coefficient] 0.9, p=0.2). The surgeon with the lowest pre-training score was the only subject to exhibit a meaningful increase in average score from 6 to 8.6 points (18 possible); the remaining surgeons had similar average scores before and after training (Figure 3).
Presence of SDM Elements in Surgeon-Patient Decision-making Before and After Training
Shared Decision-Making Element Scores of Individual Surgeons Assessed by Audiotapes Before and After Training
Patient Activation Brochure. A primary barrier to the activation brochure was a staff belief that these materials were redundant and in competition with existing patient materials. Concern was also voiced over possible additional costs associated with the production and distribution of the materials and with overloading patients with paperwork/educational material. The solution was to revise and update existing materials in a low-cost format.
Consistent dissemination of the patient activation brochure prior to the patient visit was also a barrier. Two solutions were identified. At one clinic location, the brochure was made available to patients on the counter for all orthopedic patients and targeted to spine patients. At the second clinic location, a staff person was identified who would include the brochure in the pre- visit mailing. Ideally, the activation pamphlet should be delivered prior to the visit so the patient arrives activated.
Decision Aid Dissemination. Disseminating DAs proved to be a significant barrier. Initial attempts at disseminating DAs through the new-patient mailing packet process were ineffective and subject to a number of barriers. Patient mailings to new patients included more generic materials, and the staff preparing them did not have an effective method of identifying which patients were scheduling visits due to conditions that would be appropriate for a DA. Patients sometimes did not have a condition-specific diagnosis prior to a visit, so they could not be identified for a specific DA. Concerns about additional mailing costs due to the size and weight of the DA were also raised.
In response to these barriers, a process was established where surgeons and patient care coordinators (PCCs) would be responsible for disseminating the DAs. This process was only moderately effective. These providers sometimes forgot to take aids with them to the patient visit. After the visit, surgeons operating under strong time pressures, moved on to another patient and did not go back with a DA or develop a system where other providers following up with the patient could give the patient the DA. Another problem was that patients were sometimes diagnosed and made a preference-sensitive treatment decision during the same visit, without the ability to access the DA in a timely fashion. To adequately fulfill the requirements of SDM, a patient must have the DA in a manner that allows them the time to access the materials before making a decision. Dissemination of DAs in a timely manner to assist in the DM process was a consistent problem throughout the implementation process and was not adequately solved.
Multiple Clinics. The project was piloted at two clinic locations in two different hospitals within the health care system. Varying clinic policies and procedures meant that all proposals had to go through two different processes to be approved and/or implemented. This led to delays and extra time spent seeking approval for the same intervention at two sites.
Subspecialty Buy-In. The project proposed piloting SDM in all orthopedic surgery subspecialties with preference-sensitive elective treatment options (e.g., total knee, total hip, total shoulder, spine surgery). The department chair (a spine surgeon) served as the champion for implementation of SDM throughout his department. Meetings were held to engage orthopedic surgeon subspecialty groups. Resistance from opinion-leaders in the total joint group led to a focus on orthopedic spine surgery. The process of obtaining support from the spine group was facilitated by the leadership of the department chair and participation of spine surgeons on the project team. Teams proposing wider system change should strive to have key opinion leaders of all stakeholders represented on the team. They should also demonstrate the necessity of making any system change, especially with presentation of performance data.
Decision Aids. DAs were initially developed from existing patient education materials being used by surgeons; however, the surgeons did not recognize them as a repackaging of their own patient education information. The spine group subsequently chose two commercially available DAs. Surgeons had multiple concerns with these DAs. They were concerned that a single DA describing a condition with treatment alternatives would not be appropriate for all patients, as “one size doesn’t fit all.” Surgeons also believed that treatment options are not always appropriate, and that they have the best insights into a proposed course of treatment for a patient. The surgeons disagreed with the clinical content and the scientific quality of the evidence in the DA. They believed that their outcomes were statistically different than the outcomes presented in the DA. They also were concerned that the explicit numerical estimation of risks and surgery success would unnecessarily scare patients.
Surgeon Training. Surgeon training presented a series of implementation barriers. The time constraints and scheduling demands of busy practice settings impacted training on multiple levels, including initial trainings, on-going assessments, and on-the-fly coaching. Assessment and on-the-fly coaching, which were designed to occur between visits, were often rushed or deferred in order to stay on the clinic schedule. The solutions developed were to keep the trainings short and focused and to observe and assess patient and provider interactions and provide on-the-fly coaching as time and workflow permitted. The project was successful at meeting this goal in most situations.
Support training materials were developed to address the time constraints to training and to support ongoing learning. A pocket reminder card was laminated, double-sided, and small enough to be carried in a jacket pocket, on a clipboard, or positioned at a work station to assist as both a learning aid and prompt. One side had the 10 elements in a checklist format, while the second side had the elements clustered by conceptual dimension. These conceptual dimensions were developed as another way to assist providers in learning and implementing all the elements of SDM, and they have been incorporated into an SDM “Train-the-Trainer” Toolkit.15
Posters with the elements laid out in the conceptual dimension format, as shown on the pocket reminder card, were posted in locations where surgeon and patient interactions occurred. At one clinic location, there were alcoves near the entrance to the exam rooms where patient records were placed for the visit. The posters were placed there each day so providers could pull one out with the records and review it if needed. Posters were also placed in common areas but not in exam rooms. The impact of these passive approaches was not assessed.
To also address the time limitations in training, “on-the-fly coaching” was instituted. This included trainer observation (“scorecard”) plus surgeon self-assessment. These coaching sessions were short, focused sessions that occurred in the clinic immediately following the observation of a surgeon-patient interaction. The surgeon and trainer compared their assessments, identified missing SDM elements, and discussed ways to more effectively engage in SDM in future encounters.
Our study implemented SDM in orthopedic spine clinics in two hospitals within a university health care system to improve the treatment decision process and informed consent discussions between physicians and patients. Inadequate informed consent is an underlying and significant contributor to patient dissatisfaction, complaints, and medical liability. Poor communication is correlated with patient complaints against physicians, as well as high physician liability.16 Patient complaints often arise from differences in understanding regarding risks and benefits, with a relatively large proportion of complaints (25 percent) focused on issues surrounding treatment options and alternatives.17
Shared decision-making with DAs has been shown to improve patient understanding of procedures, risks, and alternatives.18 Theoretically, a patient who is better informed of the risks, who is more satisfied with the physician-patient communication process, and who takes greater ownership of the shared decision, would be less likely to file a lawsuit in the event of a poor outcome. Washington State statute provides specific protections from lawsuits if SDM occurs.7,8 The use of patient DAs has been shown to change mock juror conclusions regarding medical negligence, with use of a DA leading jurors to nearly unanimously conclude that care met standards.19
Successful organizational change, including implementation of SDM, involves motivating the change, creating a vision adapted to local needs, garnering political support with clinical champions and stakeholders, managing the transition through coaching and data collection, and sustaining the momentum with continued resources and coaching.20,21 Studies of implementation of a surgical checklist, another formalized communication process, identified training techniques, feedback, senior clinician buy-in, and local adaptation as important facilitators of successful implementation.20 Barriers for checklist implementation included an imposed approach, resistance to change, repetition with current practice, key opinion-leader resistance, and tool content and evidence base. To maximize success, an institutional commitment of ongoing resources must be made to provide continuing support for implementation. Both structural and individual change of deeply entrenched procedures takes substantial time to accomplish.
Most studies of SDM have studied implementation within primary care.1,2,22,23,24 Effective implementation of SDM with DAs for joint replacement surgery reduced hip replacement by 26 percent and knee replacement by 38 percent, and it lowered costs by 26 percent in a managed care environment.22 Many of the barriers and facilitators for implementation of SDM in primary care were similar to ones that we found in surgical specialty care, including time constraints, high physician workload, physician training/culture, and DA distribution problems.1,2,23,24 Advocacy by leadership at all levels, cultural embracement of the financial/safety need to reduce practice variation coupled with the Washington State statute, physician training programs with feedback, and constant evaluation and iterative improvement were key facilitators of implementation of SDM.22,24 Automatic triggers for DA distribution (e.g., referral to an orthopedic surgeon for a total knee replacement) and engagement of teams other than physicians have also been suggested as solutions.1 These facilitators may not be effective when SDM is employed to improve physician-patient treatment and informed consent discussions, which require the physician to take primary responsibility for SDM. In addition, referral for total joint surgery is more straightforward than referral to see an orthopedic surgeon specializing in spine surgery. Many patients with spine disorders are not surgical candidates and lack the option to make a treatment choice involving surgery.
An important observation of our study is that physicians have deeply ingrained scripts for patient interactions that are significant barriers to learning new communication methods and incorporating new SDM techniques. Experienced physicians developed schema that guide their patient interactions to work efficiently. A barrier to disrupting schema is that surgeons often believe they are already doing SDM prior to implementation.
Our training had a limited impact on the communication schema of these physicians. Our analysis of the audiotape transcripts of surgeons’ clinical DM with patients showed minimal increase in their use of SDM elements, especially relating to the patient’s role in treatment DM. A bias currently exists in medical education about the patient’s role in making decisions that ultimately affect their care. Many physicians believe that the patient has a minor role in decisions regarding their health care treatments. This bias must be addressed throughout the system in order to effect structural change and make SDM a viable long-term success in physician/patient discussions of treatment options.
Lessons learned and processes developed during this project were incorporated into an SDM “Train-the Trainer” Toolkit. The SDM “Train-the-Trainer” Toolkit was developed to overcome training barriers and disseminate the training methodology.15 The Toolkit includes an instructor guide plus a number of “tools” to be used as part of the training process. The “tools” consist of a teaching guide, provider pocket reminder card, SDM elements cue poster, patient activation pamphlet, assessment checklist, SDM implementation recommendation and troubleshooting guide, external resource list, and an evaluation survey. The SDM Toolkit is available for download through the Association of American Medical Colleges MedEdPORTAL at www.mededportal.org/publication/9413.
Interjecting new communication protocols into existing schema requires ongoing training and a commitment to make changes. Our findings suggest a need for more extensive SDM training interventions and extended follow-up to interrupt these communication schema. Disrupting deeply entrenched scripts and schema used in practice must be a priority in an ongoing SDM implementation training process if the training is aimed at a large number of providers, rather than utilizing specially trained nurses who review SDM alternatives with patients. Training requires a multi-stage training process, with supportive training materials, to train physicians and experienced providers to change the scripts they use and to add the missing SDM components into their patient interactions. Future studies should incorporate improved training methods, including a longer period of training, and focus on both SDM elements and patient engagement techniques.
This was a cross-sectional study on the implementation of SDM. We did not implement permanent institutional structural changes to support and expand the project. No budget or resources were available for additional purchase of DAs, additional staffing, training resources, or other materials needed to maintain the ongoing practice of SDM at these locations or to expand it to other locations beyond the conclusion of the study. However, the project has been successful in describing institutional barriers and providing practical solutions to inform ongoing implementation of SDM within the hospital system.
Physicians and other health care providers do not routinely include all elements of SDM during clinical DM. Successful implementation of SDM requires a complete process to engage stakeholders, identify barriers to adoption and implementation, and develop interventions to overcome barriers that are unique to each implementation setting.
This work was supported by a grant (HS19532) from the Agency for Healthcare Research and Quality. We would like to acknowledge the Foundation for Informed Medical Decision Making and Health Dialog for their gift-in-kind of providing the decision aids used in this project at a reduced research rate.
Shawn L. Mincer, MSW; Department of Anesthesiology and Pain Medicine, University of Washington, Seattle. Michael J. Lee, MD; Department of Orthopaedics and Sports Medicine, University of Washington, Seattle. Richard J. Bransford, MD; Department of Orthopaedics and Sports Medicine, University of Washington, Seattle. Saint Adeogba, MD; Department of Anesthesiology and Pain Medicine, University of Washington, Seattle. Karen L. Posner, PhD; Department of Anesthesiology and Pain Medicine, University of Washington, Seattle. Lynne S. Robins, PhD; Department of Medical Education and Biomedical Informatics, University of Washington, Seattle. Pornsak Chandanabhumma, MPH; Department of Anesthesiology and Pain Medicine, University of Washington, Seattle. Michelle S. Lam, BS; Department of Anesthesiology and Pain Medicine, University of Washington, Seattle. Aaron S. Azose, BS; Department of Anesthesiology and Pain Medicine, University of Washington, Seattle. Karen B. Domino, MD MPH; Department of Anesthesiology and Pain Medicine, University of Washington, Seattle.
Address correspondence to: Shawn L. Mincer, MSW, Department of Anesthesiology and Pain Medicine, University of Washington, 19 NE Pacific Street BB1431, Box 356540, Seattle, WA 98195; emailsmincer@u.edu.
This publication is in the public domain and may be used and reprinted without permission, except for those items identified as protected by copyright. For these items, readers will need to contact the copyright holder for further permission to use and reproduce the materials. Citation as to source will be appreciated.
Your browsing activity is empty.
Activity recording is turned off.
See more...
1