Engagement rate

A critical performance measure is a clinic's rate of engagement for new clients. About 50 percent of clients in outpatient treatment drop out within the first month (Fishman et al. 1999), and many clients are no longer in treatment by their third, fourth, or fifth scheduled appointment. For both financial and clinical reasons, many programs monitor how effective the program and its clinicians are in engaging clients.

Because treatment programs expend disproportionate resources during the first few sessions, clients who drop out after only one to three sessions adversely affect the financial status of a program. In addition, early client dropout represents a lost opportunity to help a client and to affect the incidence of substance abuse in a community. Retention in treatment has been linked to better long-term outcomes (Fishman et al. 1999). At the first session, the client expresses an interest (regardless of the source of motivation) in participating in treatment. The client's “disappearance” before session four or five raises important questions: Why didn't the client return? What changed? and How can the program reduce the incidence of such premature disengagement from treatment?

Clinicwide engagement rate.The engagement rate at the clinic level is a simple calculation. The calculation is the total number of clients who attend the third (or fourth or fifth) treatment session, divided by the total number of clients who were admitted into treatment at an initial intake evaluation. This calculation can be completed weekly, monthly, quarterly, or annually.

Clinic engagement rate = Total number of clinic clients attending their third scheduled session/Total number of clients admitted into the clinic

The more frequently a program measures its engagement rate, the better able staff and management will be to track the performance measure. Any trend toward increased dropouts over time should lead management and staff members to examine carefully the contributing factors. If a program determines the engagement rate only annually, it will not be able to identify or respond to trends. However, if a provider does not have a management information system designed to collect and calculate engagement rates automatically, measuring this outcome on a weekly basis may be a burden. Monthly monitoring of the engagement rate may be a reasonable balance between burden and benefit.

Counselor-specific engagement rates.It is even more useful to look at how engagement rates differ among clinicians. Monitoring and reporting engagement rates to clinicians enable them to determine whether changes in their approach result in improved retention rates.

The calculation equals the total number of clients initially seen by a clinician who attended the third (or fourth or fifth) treatment session, divided by the total number of newly admitted clients assigned to that clinician.

Counselor-specific engagement rate = Total number of clients attending third scheduled session with counselor X/Total number of clients admitted into counselor X's caseload

It would be misleading to compare engagement rates for one clinic or clinician with those of another without adjusting for the case mix. Dramatic differences will be found between clinics and between clinicians, and these differences may be due largely to client characteristics rather than differences in clinic quality or clinician skill levels. For example, engagement rates (and other performance indicators and outcomes) vary depending on client characteristics and other factors:

• The clients' living arrangements (e.g., clients living in shelters tend to do less well than those with stable housing)
• Employment status
• Co-occurring conditions (e.g., clients with significant mental disorders in addition to their substance use disorder tend to do less well)
• The substance abused

By providing clinicians with information they can use to guide their improvement efforts, executives empower their staff members to improve their own performance.

Attendance rate

The clinic's attendance rate consists of the total number of treatment program sessions that are attended by clients in a given period, divided by the total number of treatment sessions that were scheduled for those clients in that period.

Attendance rate = Total number of sessions attended/Total number of sessions scheduled

As an example, if the clinic had 300 client encounters scheduled over a 1-month period, but clients actually attended only 90 treatment encounters, the retention rate would be 90 divided by 300 or 30 percent.

Administrators may find it useful to monitor attendance rates for individual counselors as well as for the clinic as a whole. Observations of who is achieving the highest and lowest attendance rates may identify treatment strategies and clinical styles that are more or less effective, counselors who need closer clinical supervision or additional training, and case mixes that are inequitable. This information should be used to help all clinicians improve, regardless of their attendance rates.

Retention rate

A retention rate indicates what percentages of clients remain in treatment.

View in own window

Retention rate = Total number of weeks clients remained in treatment/Total number of clients admitted
Step 1.	For each client who entered treatment during the period under consideration (e.g., the first quarter of the year), determine how long that client remained active in treatment. It is important to select an objective measure for “active in treatment,” such as “attended at least one or more treatment sessions within 2 weeks.” For each client admitted during the period under study, calculate the total number of weeks in treatment.
Step 2.	Add the total number of weeks clients remained in treatment and divide by the total number of clients admitted.

A clinic might be interested in knowing what percentage of clients stay in treatment for 3 months. This measure might identify a program that is successful at achieving a high engagement rate but is losing many clients after the fifth or later treatment session. For example, if 100 clients entered treatment between January 1 and March 31, the clinic would add the total number of weeks that each client remained in treatment during that period and divide that number by 100 (the total number of clients).

It may be valuable to know the differences in retention rates among groups of particular interest, such as clients referred by the criminal justice system versus all other referred clients, women versus men, or payer X versus payer Y. Such studies can have important implications for resource allocation, funding, and staffing.

Abstinence rate

Rigorous monitoring of abstinence depends on reliable tests, such as the urine drug screen, Breathalyzer™ test, or saliva test. Each test has costs associated with it, which may preclude its use. It is important for clinics to track the abstinence rate because a clinic could have extremely high engagement or retention rates while its clients are still using substances. Abstinence can be measured easily, so long as objective measures, such as urine drug screens, are being used with some frequency.

The abstinence rate is arrived at by dividing the total number of negative test results obtained during a specified period by the total number of tests administered in that period. If during January a clinic administered 200 urine drug screens and 88 tested negative, the abstinence rate would be 88 divided by 200 or 44 percent.

Clinic abstinence rate = Total number of negative test results/Total number of tests administered

In monitoring abstinence rates, it is best to apply a specific timeframe to the client group being assessed. For example, a clinic may wish to calculate abstinence rates only for clients who have been in treatment at least 2 weeks. It may take that long for clients to begin achieving abstinence and for most drugs to clear from their systems. (Marijuana is eliminated from the body slowly, so clients who have been abstinent and in treatment for less than 1 month could still test positive.) Together clinic staff and management need to develop abstinence rate timeframes appropriate for their facility. One approach is to compare the abstinence rates of clients who have been in treatment for 2 weeks with those who have been in treatment for 6 weeks or longer. Abstinence rates should increase with more time in treatment.

Drug screens should be administered consistently to all eligible clients. For example, if a clinic gives drug screens only to clients who are doing poorly, the clinic will have abstinence rates that reflect the performance of its most challenged (and challenging) clients. If drug screens are given to all clients equally, the abstinence rate obtained will reflect more accurately the clinicwide abstinence rate. (For more information on drug screens, see appendix B in TIP 47, Substance Abuse: Clinical Issues in Intensive Outpatient Treatment [CSAT 2006b ].)

For some clinics, the costs of administering drug screens may be prohibitive. Although drug screens objectively measure abstinence, self-reported abstinence can be useful under certain conditions. The accuracy of self-reported data varies depending on the consequences associated with reporting current substance use (Harrison 1997). For example, a client will underreport drug use if use can result in being returned to jail, losing custody of a child, or being terminated from employment. Although self-reported abstinence alone is a less than ideal measure of abstinence, for many treatment programs it may be the only basis available for an abstinence outcome measure.

For long-term rates, self-reported abstinence may be determined during a followup telephone interview, perhaps 6 months after discharge. If the followup call is made by the client's former counselor, the client may be more reluctant to admit use than if the call is made by a staff member or researcher with whom the client has no history. Clients often do not wish to disappoint their former counselor by acknowledging that they are having difficulty and have relapsed.

Quality-of-life indicators

Problem-specific monitoring.It is important to know whether treatment has not just influenced clients' substance use problems but also positively influenced other areas of their lives.

Problem-specific monitoring may be particularly important if the mission or funding of the clinic is associated with behavioral domains. For example, a treatment facility connected with Treatment Accountability for Safer Communities or a drug court might be interested in the extent to which its program is reducing clients' criminal activities. A treatment program also might be concerned with whether its interventions are reducing behaviors that put clients at high risk for contracting infectious diseases. In either case, assessments might be administered at different points during the treatment process and after discharge to see how well clients are functioning and to track changes in behavior or status.

A program might track information needed by its funding or referral sources (e.g., drug court). Improvements in clients' employment, education, and family relationships can be important to funders and the public. The more a program is able to document the positive effect of its efforts, the better it will be able to justify its funding and argue for additional funding. It is most impressive if a program is able to establish that treatment still is having an effect several months after a client's discharge. But the followup monitoring required to obtain these data is more expensive and difficult to do than monitoring while the client is in treatment.

Support group participation.Involvement in support groups, such as 12-Step programs and other mutual-help groups, is another way of measuring continued sobriety and a client's determination to remain abstinent. Followup calls may include questions about the number of support group meetings a client has attended in the previous week or month, whether the client has spoken with his or her sponsor in the previous month, and whether the client has a home group. Programs can monitor whether their efforts lead to improvements in these important performance indicators by quarterly or biannually assessing the rate of self-reported meeting attendance.

Other quality-of-life indicators.The following quality-of-life indicators often are included on existing statewide databases and can be monitored with varying degrees of difficulty:

• Reductions in arrests, convictions, and incarcerations
• Reductions in hospitalization for mental illness
• Increased participation in afterschool programs
• Decreases in school dropout rates
• Reductions in use of welfare benefits and food stamps and in open child welfare cases
• Reductions in emergency room visits and other hospitalizations
• Increases in employability; increases in wages and number of days worked
• Reductions in social costs caused by intoxicated drivers and lost workdays
• Increases in the rates of birth of healthy, drug-free babies
• Improvements in school participation

These indicators usually are not generated on an individual clinic level but are of interest to many stakeholders.

Client satisfaction

For decades, businesses and industry have focused on measuring customer satisfaction, and this information can be valuable for OT programs. Client satisfaction provides information about the performance of both individual staff members and the clinic as a whole. For example, increasing client satisfaction may be a way to increase treatment engagement, attendance, and retention. In addition, health service providers, including treatment providers, increasingly are called on to monitor client satisfaction. Client satisfaction data point to possible causes and solutions for substandard performance. Surveys showing that clients are dissatisfied may help staff members and managers understand why retention or even abstinence rates have decreased. No nationally recognized client satisfaction survey currently exists for substance abuse treatment providers. Appendix 6-A presents a client satisfaction form that has been designed specifically for use with IOT clients. Client satisfaction forms usually are divided into three sections:

• Client satisfaction with clinic services, such as client education materials, counseling groups and educational sessions, adequacy of the facility, individual attention, and overall benefit of treatment. Clinic administrators and staff members should decide which items to include, choosing those perceived as most important to the quality of care and those of greatest concern in their service delivery.
• Client satisfaction with the counselors, which asks the client to identify his or her counselors by name and provide feedback on their effectiveness. Areas to explore include each counselor's warmth, empathy, insight, knowledge and competence, attentiveness, and responsiveness. Staff members and clients should help select and word the criteria by which counselors will be rated.
• Confidential descriptive (demographic) information about the respondent, including age, gender, ethnicity, and sexual orientation. The information from this section can help a clinic determine such factors as whether women are more satisfied with a program than are men or whether clients with a disability are as satisfied with treatment as are other clients. The survey should be administrated to a specific client population. Programs can use the findings to guide changes in training, staffing, or programming.

Satisfaction of referral sources

Conducting a structured telephone interview with the program's key referral and funding sources at 3-month intervals can elicit considerable information about how the program and staff are viewed. Such calls can be a check on whether the program is providing each referral source with the information the agency needs in a timely, helpful fashion. The interviews can identify areas of complaint or potential friction before difficulties or misunderstandings escalate into problems. These telephone calls also can be used to explore new opportunities for expanding or refining services. (See appendix 6-B for a sample form.)

Success of client transfer

An important measure of a program's effectiveness is the percentage of clients who have transferred successfully to and been retained in long-term, low-intensity outpatient services following completion of an IOT program.

Client dropout rate

Another valuable approach to performance improvement is to conduct studies of clients who have dropped out of treatment. Because early treatment sessions are the most expensive, clients who drop out of treatment represent, in many ways, the greatest loss to a program. A study designed to understand better who drops out of treatment and why can help guide changes in the program that ultimately yield great benefit.

To conduct such a study, the program can conduct telephone interviews of the last 50 or 100 clients who dropped out of treatment. The interviews should be done by an independent (noncounseling) staff member, such as a student intern or an assistant. The caller states that the purpose of the call is to

• Determine what factors led the client to leave treatment
• Gather information for a program evaluation
• Help the program be more effective and responsive to the needs of its clients

One result of open-ended interviews is that patterns of comments often emerge. A preponderance of similar responses can indicate that changes are needed. For example, a program whose client population was overwhelmingly male conducted a study of women who had dropped out. The study confirmed that the women had dropped out because group sessions were dominated by male viewpoints, and the women felt their concerns were not being addressed.

When conducting a dropout study, the caller should include an invitation to each client to return to treatment. The invitation may be all that is needed to reengage a client in the recovery process.

Addiction Severity Index

The Addiction Severity Index (ASI) is a useful outcome measurement tool (McLellan et al. 1992b ) that helps assess a client's treatment needs. Free copies of the ASI and guidelines for using it can be downloaded from www.tresearch.org. The ASI is a standardized instrument that has good reliability and validity and can be used to collect information for comparison across sites and at different points in time.

Trained staff members should administer the ASI

• On client intake to provide baseline status and aid in treatment planning
• At meaningful intervals during treatment to measure progress (e.g., after 3 months)
• At discharge to assess outcomes in seven key areas of a client's life, including recent substance use and legal, occupational, medical, family/social, and psychiatric statuses
• At 3 months after discharge to measure long-term change in the client's status and behavior

The ASI provides programs with

• A subjective client evaluation and an interviewer evaluation of problem severity; both can provide indications of client treatment priorities
• An easy-to-use monitoring tool for clinical supervision; severity scores in each problem domain can be checked quickly to ensure adequate treatment planning
• The capability to monitor outcomes of discharged clients through followup; clients can be reevaluated using the ASI followup interview
• A research tool to derive objective and subjective measures of need and improvement
• A rich database through which issues such as the relationship between client characteristics and outcome data can be examined
• A standardized assessment instrument that permits comparisons across programs and levels of care

Risk Assessment Battery

A frequently used measure for risk of infectious disease is the Risk Assessment Battery (RAB), which is self-administered. Monitoring of risk reduction for infectious diseases might involve administering the RAB at intake, after 2 months of treatment, at discharge, and then 1 to 3 months after discharge. (Visit the Treatment Research Institute Web site at www.tresearch.org to download this instrument.)

Handling data in a confidential and sensitive manner

It is important to show sensitivity toward staff by handling data confidentially. This usually is done by presenting only clinicwide data—not data on individual performance—to the staff and the public at staff meetings or in reports to funders. For example, an administrator might discuss changes in risk-reduction measures at the level of the clinic, not for individual therapists.

Types of comparisons

It is natural and, under some conditions, beneficial for staff members to compare their performance with that of other staff members. However, counselors achieving the highest performance rates may be scoring well because of experience, training, case mix, random fluctuation, or unique talent. The goal is to help every counselor in the clinic improve over time. In other words, a counselor whose engagement rate has been 30 percent should be acknowledged for increasing the rate to 50 percent (even though the average rate in the clinic is 60 percent). Comparing a counselor who has a low engagement rate to the clinic average can lead to discouragement and even poorer performance (Kluger and DeNisi 1996). Such comparisons should be avoided. Administrators should focus on clinicwide data and improvement initiatives. Counselor-specific data should be released confidentially to individual counselors.

Strategies to encourage staff members' improvement

Effective strategies to improve performance include

• Providing individualized confidential feedback about performance
• Tracking changes in performance over time

Kluger and DeNisi (1996) reviewed more than 2,500 papers and 500 technical reports on feedback intervention conducted in a variety of settings. They noted that performance feedback interventions are most effective if the feedback is provided in an objective manner and focuses on the tasks to be improved. Feedback should address only things that are under counselors' control. Interventions that make the feedback recipients compare themselves with others can result in worse performance. (Data on the individual performance of counselors should be confidential and secured; these data can be presented as counselor A, B, C, etc.)

Feedback data can be used to encourage staff members who have shown exceptional improvement. Identifying a staff member of the month can be an incentive for achievement. The key is to recognize improvement publicly, based on objective data. This kind of recognition encourages new staff members to learn from their high-performing colleagues. Those who are performing consistently at the highest levels (known as positive outliers) can be acknowledged formally. These high achievers can be invited to give presentations, provide training, or recommend ways to improve the organization's performance. Under certain circumstances, arrangements can be made for counselors to observe productive counseling sessions. This kind of recognition should not be made too quickly; it should be based on at least 3 months to a year of performance data. Case mix could account for a counselor's consistently superior performance (e.g., the counselor treats the highest functioning clients).

Sharing the performance results

Once a performance improvement system is in place, the findings provide important information for three groups:

• For the clinic, information about overall performance
• For the counselor, information about individual performance
• For the program's funding sources and other groups in the community, information about overall clinic performance

What should a program do if the monitoring system identifies less than satisfactory performance? This can be interpreted as good news in many ways. Knowing a problem exists is the first step in solving it. Clinic management can introduce interventions to improve performance, and the data collected will allow the effectiveness of the new interventions to be monitored. Moreover, because many programs do not have a rigorous performance improvement system, a program can distinguish itself by having these data—a potential advantage in securing funding.

Performance monitoring can reduce the frequency of disciplinary job actions (e.g., terminating someone for poor performance) because it focuses on objective measures. Performance monitoring results can lead to program changes, reallocation of resources, targeted training, and skills development. More important, when objective data are made available to all staff members, improvements may occur without any additional intervention—simply because people generally want to perform well (Deming 1986).