Inherent Quality of Speech and Language Instruments

Publication and Documentation

In the spirit of evidence-based practice, we note that publishing the information called for above in peer-reviewed journals is also critical. As noted in Chapters 2 and 4 of this report, we relaxed what might be regarded as the commonly followed standards of systematic reviews by including substantial amounts of gray literature in the form of users' manuals and instruction guides. That is, we learned in doing this evidence report that data on the reliability and validity of speech and language assessment instruments are rarely found in the peer-reviewed literature; such information is largely confined to the instrument manuals, which do not undergo a peer-review process and cannot really be considered acceptable venues for documenting these properties of the instruments. Thus, we suggest that journal editors in fields concerned with speech and language disorders encourage the submission of reports on instrument reliability and validity, identify peer reviewers who are qualified to evaluate the rigor of these types of reports, and then publish such data in their journals.

Normative Data and Samples

Instruments developed to measure speech and language disorders in adults have used samples of individuals with disorders as the normative sample. Unquestionably, finding the numbers of individuals with disorders needed for the development of psychometrically sound assessment instruments is challenging indeed. Nonetheless, developers report almost no evidence regarding the representativeness of their samples used in standardizing these instruments. This issue needs attention in future research.

A related issue is the importance of developing normative data for subgroups and avoiding inappropriate aggregation of subgroups defined by different disease processes, comorbidities, or prognoses. For example, in the standardization of adult language instruments, individuals with traumatic brain injury and those with cerebrovascular accidents have been grouped together. Patterns of cognitive-linguistic functioning following these two types of neurological insults are different, however, and clinicians' ability to interpret assessment results is impeded when normative databases combine data on these two subgroups.

An issue with several measures of child speech and language development reviewed for this report pertains to the systematic exclusion of children with disabilities from the normative sample, resulting in a lack of representativeness of the samples and possible inflation of the numbers of children who would be identified with speech or language disorders using these measures. In short, the representativeness of samples used in instrument development and standardization is an area for attention in future research across the age span and various disorders comprising the focus of this report.

Costs, Benefits, and Harms

Assessing the feasibility of use of these instruments in clinical settings was beyond the purview of this evidence report. Consequently, we did not formally evaluate the cost or the burden on patients, clinicians, or administrators of actually applying these instruments. (Chapter 3 and its instrument-specific tables describes the basic steps for using the instruments and comments on usability.) Information on costs and burden to patients and to those in health care delivery settings might well be valuable, however, in helping SSA or clinicians in selecting among otherwise seemingly similar instruments.

In addition, in extreme cases of impairment, using the types of instruments reviewed in this report may be an unnecessary cost, with disability status being readily determined on the basis of gross clinical criteria that do not require fine-grained assessment or quantitative comparison to normative samples. Thus, areas for future research are (a) to compare the relative sensitivity and specificity of different approaches to disability determination for different types and degrees of speech and language impairment and (b) to determine when the relative costs and benefits justify the addition of standardized instruments to the assessment process rather than relying solely on clinical judgments.

Also not dealt with in this report or literature on this topic is the matter of the adverse effects or harms of diagnostic testing or disability evaluation. Although clinicians, SSA experts, and others may not judge harms stemming from the use of these instruments to be large, nonetheless the possibility exists that patients may be misdiagnosed or categorized wrongly. Persons with true disease may be missed and, thus, wrongly kept off or struck from SSA disability eligibility rolls; persons identified as disabled who truly are not may suffer emotionally or financially from such mislabeling. In both cases, SSA faces the twin problems of inappropriate persons covered, or not covered, within the disability program. We urge that researchers take a broader perspective to the investigation of speech and language instruments, so as to shed some light on the likelihoods that adults or children may be mislabeled (in both positive and negative ways) and on the consequences of such labeling.

Normative Data and Subpopulations

With the increasing cultural, linguistic, and racial diversity of the US population, the applicability of assessment instruments to individuals who are members of different subpopulations is of crucial importance to clinical diagnosis and the process of disability determination. In addition to demographic subpopulations, the applicability of speech and language assessment instruments for reliable and valid assessment of individuals with different disorders is important, because speech and language impairments may contribute to the disability of people identified with other disorders, such as severe physical impairment, mental retardation, learning disorders, and hearing impairment. Inclusion of representative numbers of subpopulation members in normative samples during instrument standardization is important, but it is not sufficient to ensure the applicability of instruments to various subgroups.

Improving the evidence base requires analyses examining reliability and validity of instruments for subpopulations, not just for the total normative sample. The only instruments we reviewed that have provided good quality evidence in this respect were the Test of Language Development (TOLD)-Primary, 3^rd Edition and the Test of Language Development-Intermediate, 3^rd Edition. Subgroups for which TOLD information can be found include children who are speech-language disordered, learning disabled, mentally retarded, and hard of hearing. All three adult language instruments were standardized on individuals who have language and cognitive impairments associated with strokes and traumatic brain injury, but as noted earlier, the data from these two groups were combined and analyses for subgroups were not presented.

Despite this start on instrumentation for various subgroups, clinicians and policymakers need to recognize that dialect, language, or cultural differences, or functional differences due to certain types of impairment, may well preclude reliable and valid assessment with existing instruments. Despite the existence of a large number of speech and language assessment instruments, we still lack appropriate instruments for reliably and validly assessing speech and language in many of these diverse subgroups. Thus, future research funding and priorities should be directed at addressing these serious deficiencies. Funding sources should encourage research teams that represent collaborations among professionals with expertise in speech and language disorders, cultural experts for the demographic subpopulations of interest, professionals with expertise in disorders that often co-occur with speech and language impairment, and psychometric experts.

Evaluating Existing and Emerging Therapies and Treatments

Some work in the evidence-based practice field (for instance, that done on behalf of the current US Preventive Services Task Force¹²⁰) tends to examine clinical questions about diagnostic (or screening tests) within a context of effective therapies. We did not take that approach in this evidence report because of the already expansive scope of the assignment; hence, questions concerning the efficacy or effectiveness of various treatment of speech and language disorders have here gone unexamined.

We note, however, that a rich agenda of research remains to be pursued concerning appropriate ways to manage speech, language, or voice disorders in both adults and children. A necessary part of such investigations involves tracking patients' progress over time, and obviously the types of instruments reviewed here could play a part in such outcomes assessments.

We caution, however, that the deficiencies in many of these popular and well-known instruments must be addressed before they can be used with confidence in treatment trials or studies. The basic measurement issues were discussed above, but in addition, methodologic work needs to be done on the responsiveness of these instruments (that is, on their sensitivity to change and on the calculation of appropriate effect sizes that reflect change over time for individuals and groups). One strategy for those engaging in or supporting research on the management of patients with speech and language disorders is to build solid methdologic research directly into treatment and rehabilitation studies, thereby strengthening both the given studies and the measurement field as a whole.

Research on treatment efficacy or effectiveness is also relevant to the concerns of the SSA in disability determination. One factor that must be considered in developing a prognosis for a patient's functioning at the end of a 12-month (or longer) interval is the expected response of the patient to intervention. Thus, building the evidence base on treatment outcomes in speech and language disorders will contribute to policy development and more informed clinical decision-making in this area.

Impairment and Disability

Most of the instruments we reviewed were designed to provide a measure of the type or degree of impairment (or both) that an individual experiences. The literature has a dearth of information on the relationship between the type or degree of impairment and the functioning of the person in any usual life activities, including those of concern to the SSA in its disability determinations. This "real world" functioning question suggests a rich research agenda that would not only assist the SSA in decisions about disability but also contribute to the "ecological validity" of all speech and language assessments. We need both more instruments providing direct measurement of activity limitations and participation restrictions and more research demonstrating the relationship between speech and language impairment and activity limitations or participation restrictions.

This is not a completely unexplored area in speech and language instrumentation. For example, the Functional Assessment of Communication Skills for Adults was developed to evaluate the ability to communicate in daily life activities despite existing speech, language, cognitive or hearing impairment.²⁴ Our national panel of experts chose instruments based in part on the likelihood of finding an evidence base in the peer-reviewed literature. Thus, the bias toward instruments measuring impairment in the current report reflects the fact that expert measures of speech and language functioning are very limited. The interests of current policymakers and clinical practitioners may be better served by investing resources into refining such measures and developing an evidence base for them rather than or in addition to focusing on measurement of impairment.

Prediction of Future Functioning

We do call for further research on the second key question considered in this report, that is, the ability of speech and language assessment instruments to predict future functioning or performance. As discussed in Chapters 3 and 4, we found very limited evidence in the literature related to this topic. Predicting future functioning is a key criterion for disability determination, so the SSA will need research results that document the ability of these measures to provide robust, longer-term predictions. Said another way, evidence must be relevant to predicting how an individual will function at least one year from the time of the initial disability determination.

Research on this question needs to be large scale. Moreover, it should not be limited to the predictive value of instruments in assessing specific intervention programs or in predicting future performance of a restricted subgroup. Rather, in terms of broad nation concerns about disability, research should consider prediction of future test performance and future adaptive performance in everyday life among key subpopulations reflecting age, language, co-existing conditions, and other factors.