Appendix D: Methodology

Andrea K Biddle; Linda R Watson; Celia R Hooper; Kathleen N Lohr; Sonya F Sutton

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Biddle AK, Watson LR, Hooper CR, et al. Criteria for Determining Disability in Speech-Language Disorders. Rockville (MD): Agency for Healthcare Research and Quality (US); 2002 Jan. (Evidence Reports/Technology Assessments, No. 52.)

This publication is provided for historical reference only and the information may be out of date.

This publication is provided for historical reference only and the information may be out of date.

Criteria for Determining Disability in Speech-Language Disorders.

Show details

Contents

< Prev Next >

Appendix D: Methodology

This appendix provides additional detail on selected aspects of the methodological approach adopted by the RTI-University of North Carolina at Chapel Hill (UNC) Evidence-based Practice Center (RTI-UNC EPC). We first discuss the process we used to modify the key clinical questions and the causal pathway. We then document how we selected instruments and set priorities for literature search and evidence review. We end the appendix with a discussion of the supplemental analysis of the usability of the selected instruments and their manuals. Tables D1 through D4 and Figures D1 through D6 presented here supplement the text in Chapter 2, its Methods Appendix, and this Appendix.

Revision of Key Clinical Questions and Causal Pathway

We developed preliminary key clinical questions and causal pathway in response to the initial request for proposal from the Social Security Administration (SSA) and the Agency for Healthcare Research and Quality (AHRQ). To refine these conceptual issues, we organized a one-day meeting (September 18, 2000, in Rockville, Maryland) to

Solicit input from the meeting participants on the utility and appropriateness of the causal pathway and refinement of key clinical questions, and
Identify and prioritize evaluation tools to be included in the evidence analysis.

Meeting participants included the 10 members of our formal Technical Expert Advisory Group (TEAG) (see Appendix B), AHRQ staff, and appropriate SSA representatives. The meeting participants (Table D1) included individuals with clinical expertise in speech, language, and voice disorders in adults and children, neurology (adult and pediatric), otolaryngology, developmental pediatrics, and both educational and vocational aspects of speech and language disorders; representatives of professional societies (e.g., American Psychological Association, American Academy of Otolaryngologists-Head and Neck Surgeons, American Speech-Language-Hearing Association) and health care systems also participated.

Table

Table D1. Participants in September 18, 2000 Meeting.

Key Clinical Questions

During the meeting the discussion of the key clinical questions concerned (1) whether language impairment included oral, aural, and written impairments; (2) which populations to include or exclude; and (3) differences between impairment and function and whether it is appropriate to use tools for impairment to evaluate future functioning or performance. The first issue was addressed by leaving the terminology in the key clinical questions broad (i.e., speech and language disorders) rather than by limiting to spoken language (current SSA criteria examine spoken or verbal rather than written language). The second issue was not resolved; rather, the RTI-UNC EPC evaluated tools looking for evidence for all of the populations listed. The final issue of using measures of impairment to predict future functioning was addressed by rewording the second key clinical question to ensure that measures of impairment are used to predict future impairment and that functioning/performance tools are used to predict future functioning/performance.

Causal Pathway

During the meeting, participants suggested minor revisions to the causal pathway. Specifically, we included voice impairments as a separate category of impairments and added responsiveness (i.e., the ability of the test to detect changes in impairment and to differentiate between levels of severity) and appropriate normative data to the relevant test characteristics of the evaluation tools. We made the former change because voice production is a separate issue with different evaluation tools and complexities. The latter change was made to describe more comprehensively the characteristics of evaluation instruments.

Selection and Prioritization of Instruments for Review

During the September 18, 2000 meeting, we asked participants to select and prioritize instruments for review. The scientific director provided meeting participants with a partial list of speech-language diagnostic tools to use as a reference during this process. The list was not exhaustive but rather was designed to serve as a trigger for suggesting tools. After some discussion of the SSA's inability to accept tests for which normative data are not current, the participants were reminded that the purpose of the task was to select tools for which evidence is available and to allow the SSA to use the resulting evidence report to develop criteria.

The selection process began with the EPC study director and Center co-director encouraging meeting participants to nominate evaluation tools in each of five categories -- adult language, child language, adult speech, child speech, and voice. They then solicited a single tool from each individual, going around the table until no participants made additional suggestions. In general, TEAG members suggested the majority of the evaluation tools. Meeting participants generally suggested only tools in their areas of expertise; physicians were less likely to contribute tools during this process.

In all, 39 separate instruments emerged. We could not have conducted systematic literature reviews and evidence analyses for each of the 39 tools elicited given the project timeline and resources, so the EPC co-director asked meeting participants to set priorities for the tools within the five categories, selecting three tools in each. A formal voting process was not used to elicit the priorities, but the participants substantially agreed about the tools finally selected for review.

During the prioritization process, meeting participants articulated several principles for guiding instrument selection. For language disorders, participants suggested that tools needed to represent receptive language, expressive language, and functional language, with emphasis on tools that test language disorders broadly rather than a particular aspect of language. The panel did not suggest several instruments considered to be standards in the field (e.g., ASHA Functional Assessment of Communication Skills²³) because reliability and validity data, although available, evidently had not been published in the peer-reviewed literature. For child language conditions, participants also considered it important to select tools that could be used for different ages groups (i.e., 0 to 3 years of age, 3 to 5 years, and school age). For speech, tools evaluating connected speech were given greater consideration that those evaluating only single word production. Meeting participants also attempted to balance tools for elicited behaviors with those evaluating observed behaviors.

As noted, meeting participants suggested 39 instruments during this process -- 10 for adult language, 15 for child language, five for adult speech, six for child speech, and three for voice disorders. Table D2 gives the entire list, with those selected for review indicated in italicized, bold text. More than three tools may be indicated for review if the tools apply (in various forms) to both adults and children or appear in both English and Spanish and would likely be captured in a single literature search. As described in Chapter 2, we subsequently excluded phonological process analysis after consultation with TEAG members in December 2000 and with colleagues in the Division of Speech and Hearing Sciences at UNC-Chapel Hill.

Table

Table D2. Tests Selected by Meeting Participants by Disorder Category^a.

Supplemental Analysis -- Usability Analysis

When deciding which instrument to use, a clinician must evaluate whether the manual provides sufficient information on how to administer and score the instrument. As part of our analyses, we evaluated the usability of the instrument manuals. Two second-year speech and language pathology graduate students in the Division of Speech and Hearing Sciences at the University of North Carolina at Chapel Hill independently evaluated each manual for comprehensiveness and ease of use. Each has completed a minimum of 80 hours of supervised training in speech and language disorder assessment divided equally between adults and children. Thus, the evaluations of these raters represent what we might be expect if an experienced speech and language pathologist used an unfamiliar instrument for the first time.

Each reviewer independently completed the Usability Evaluation Form (Figure D6) supervised by EPC clinical experts. After the evaluations were complete, we entered the data into a Microsoft Excel^® spreadsheet, coding a "1" if the rater indicated that the instrument met the criterion (i.e., circled "yes" on the form), "0" otherwise. To assure consistency between the graduate student reviewers, we computed Cohen's kappa statistic of inter-rater reliability³⁵ using SAS, version 6.12 (The SAS Institute, Cary, NC) and percentage agreement between the raters (Table D4). For each instrument, we report the number and percentage of criteria met and describe where the two reviewers disagreed in their assessments. We calculated these statistics by adding the number of criteria met, scoring 1 point for criteria that both reviewers rated as being met and 0.5 points for criteria where only one rated it as being met; and dividing by the total number of criteria (i.e., 8).

Table

Table D4: Percentage Agreement and Inter-rater Reliability (Kappa)Appendix D: Methodology.

Kappa values for the individual criterion ranged from 0.34 to 1.00, suggesting slight moderate to almost perfect agreement.³⁹ Inter-rater agreement ranged from 76.5 percent (13/17) to 100% (17/17). We re-reviewed the criteria for which we observed the lowest level of agreement. In most cases, the disagreement amounted not to whether instrument met the criterion but how much information the manual provided to the reviewer. One of the reviewers seemed to require more detail on use to be comfortable with the instrument and, thus, to rate the criterion as having been met.

Table D1. Participants in September 18, 2000 Meeting

Table D2. Tests Selected by Meeting Participants by Disorder Category^a

Table D3. Search Terms Employed in the Literature Review

Table

Table D3. Search Terms Employed in the Literature Review.

Table D4: Percentage Agreement and Inter-rater Reliability (Kappa)Appendix D: Methodology

Figure D1. Evaluation Instruments for Speech and Language Disorders Abstract Review Form

Article Author:____________________________________________________________________

Journal:_________________________________________________________________________

Year of Article:____________________________________________________________________

UID (Unique Identifier):______________________________________________________________

Name of Tool:_____________________________________________________________________

Database: Circle one of the following:

MEDLINE CINAHL PSYCHLIT ERIC HAPI UNKNOWN

Abstractor Initials:______________________________

If the abstract is not available, stop here and return this form to Anne Jackman.

1.	Includes information on: reliability (e.g., internal consistency, test-retest, intra- or inter-rater) AND/OR validity (e.g., construct, concurrent, or predictive validity for future communicative functioning) of evaluation tool(s)

Yes

Cannot Determine

2.	After completion of study each analysis group is greater than or equal to 20 subjects

Yes

Cannot Determine

3.	After completion of study each analysis group is greater than or equal to 10 subjects

Yes

Cannot Determine

Study design is one of the following:

RCT (double, single-blinded or cross-over)
Nonequivalent control group design
Prospective or retrospective cohort
Cohort study not otherwise specified
Case-control study
Psychometric testing of all types
Meta-analysis, meta-regression, or cross-design synthesis

Yes

Cannot Determine

5.	Includes children (birth-21) and/or adults*

Yes

Cannot Determine

: * May include older individuals but majority must fall into this age range.

IF ANY ITEM IN THE GRAY AREA IS CIRCLED, THE ARTICLE IS EXCLUDED.
IF ANY ITEM IN "CANNOT DETERMINE" AREA CIRCLED, PULL ARTICLE FOR FURTHER REVIEW (PFFR).
INCLUDE:__________	EXCLUDE:___________	PFFR:______________

Figure D2. Data Extraction Form for Peer-Reviewed Articles-Criteria for Determining Disability in Speech Language Disorders (pages 305-312, following)

DATA EXTRACTION FORM FOR PEER-REVIEWED ARTICLES CRITERIA FOR DETERMINING DISABILITY IN SPEECH LANGUAGE DISORDERS
SECTION I: Inclusion/Exclusion Checklist
A. Date of Review (MM/DD/2001): ____ / ____ / 2001
B. Tool Used: (specify instrument(s)/version(s) tested first, criterion/reference test next)
Instrument #1: _____________________________________ Version: ___
Instrument #2: _____________________________________ Version: ___
Instrument #3: _____________________________________ Version: ___
Instrument #4: _____________________________________ Version: ___

: * May include older individuals but majority must fall into this age range.

C. ABSTRACTOR: BEFORE BEGINNING TO ABSTRACT THE ARTICLE, ANSWER THE FOLLOWING QUESTIONS.

1. Includes information on reliability (e.g., internal consistency, test-retest, intra- or inter-rater) AND/OR validity (e.g., construct, criteria, or predictive validity for future communicative functioning) of evaluation tool(s)	Yes	No
2. After completion of study each analysis group is greater than or equal to 20 subjects	Yes	No
3. After completion of study each analysis group is greater than or equal to 10 subjects	Yes	No
4. Study design is one of the following: RCT (double, single-blinded or cross-over) Nonequivalent control group design Prospective or retrospective cohort Cohort study not otherwise specified Case-control study Psychometric testing of all types Meta-analysis, meta-regression, or cross-design synthesis	Yes	No
5. Includes children (birth-21) and/or adults (18-62)*	Yes	No

: * May include older individuals but majority must fall into this age range.

ABSTRACTOR: IF ANY ITEM THE GRAY AREA IS CIRCLED, THE ARTICLE IS EXCLUDED. RETURN ARTICLE AND FORM TO STUDY DIRECTOR IMMEDIATELY.
SECTION II: Study Background Information
1. Date of Review (MM/DD/2001): __ __ / __ __ / 2001
2. Tool Used: (specify instrument(s)/version(s) tested first, criterion/reference test next)
Instrument #1: _____________________________________ Version: ___
Instrument #2: _____________________________________ Version: ___
Instrument #3: _____________________________________ Version: ___
Instrument #4: _____________________________________ Version: ___
3. Population (highlight one):	1= Children <=21 yrs	2=Adults <= 62 yrs
4. Disorder (highlight one):	1=Not reported	2=Unclear	3=Language
	4=Speech	5=Voice
	6=Other (specify)_______________________________
5. Year Published (YYYY):___ ___ ___ ___
6. Country where study conducted: __ __
1=United States	2=Canada	3=Britain/United Kingdom
4=Australia	5=New Zealand	6=European Country_______________________
7=South America_____________________		8=Asia______________________________
7. Number of Authors: __ __
8. Surname of First Author_______________________________________
9. Background of First Author (highlight all that apply):
1=Not reported		8=Pediatrics
2=Unclear		9=Neurology
3=Speech Language Pathology		10=Otolaryngology
4=Audiology		11=Education
5=Psychology		12=Test agency/publisher
6=Neuropsychology		13=Other (specify)________________________
7=Psychiatry
10. Funding Source (highlight all that apply):
1=Not reported		5=Industry
2=Unclear		6=Government Agency
3=Consumer/Patient Organization		7=Professional Organization
4=Charity		8= Other (specify)_____________________
SECTION III: Study Design and Description
12. Main Objectives (as described by author, give page and paragraph number in article):
Page:_________	Paragraph(s):___________
____________________________________________________________________________
____________________________________________________________________________
____________________________________________________________________________
____________________________________________________________________________
13. Study Design (highlight one only):
1=Not reported		6=Prospective cohort (w/out comparison group)
2=Unclear		7=Retrospective cohort (w/ comparison group)
3=Controlled trial, randomized		8=Retrospective cohort (w/out comparison group)
4=Controlled trial, non-randomized		9= Other (specify)_________________
5=Prospective cohort (w/ comparison group)
14. Length of Study:___________________________________________________________
(If follow-up times vary, include the range, mean, median, and/or mode if they are given.)
15. Study/Evaluation Setting (highlight all that apply):
1=Not reported		7=Speech/Language Clinic: Inpatient
2=Unclear		8=Speech/Language Clinic: Outpatient
3=Community		9=Hospital: Inpatient
4=School: Preschool		10=Hospital: Outpatient
5=School: Elementary School		11=Other (specify)____________________
6=School: Middle/High School
Outcome(s) Measured
16. Is outcome measured >= 12 months after initial measurement?
1=Yes
2=No	→Go to next question
3=Not reported	→Go to next question
4=Not clear	→Go to next question
17. Is outcome measured >= 6 months after initial measurement?
1=Yes
2=No	→See Study Director immediately
3=Not reported	→See Study Director immediately
4=Not clear	→See Study Director immediately
18. Outcome Measure(s) (i.e., what types of functioning/performance, instrument used to measure):
(Abstractor: Add or eliminate as needed)
Outcome #1:__________________________
Instrument/Tool #1:_____________________		Version: _______________
Outcome #2:__________________________
Tool #2:_____________________________		Version: _______________
Outcome #3:__________________________
Tool #3______________________________		Version: _______________
Characteristics of Assessors/Evaluators
19. Number of Assessors/Evaluators (if given): __ __
20. Background of Assessors/Evaluators (highlight all that apply):
1=Not reported		7=Psychiatrist
2=Unclear		8=Pediatrician
3=Speech Language Pathologist		9=Neurologist
4=Audiologist		10=Otolaryngologist
5=Psychologist		11=Graduate Student: (specify discipline)
6=Neuropsychologist		→_______________________
Characteristics of Patients/Subjects
Abstractor: Add groups as required.
21. Defining Characteristic(s) (e.g., children with SL impairment vs. normal, age-matched children; Wernicke's vs. Broca's aphasia, etc.):
Group 1: _______________________________________________________________
Group 2: _______________________________________________________________
22. Comorbidities (e.g., learning disorders, mental retardation, hearing impairment, etc.):
Group 1: ____________________________________________________
Group 2: ____________________________________________________
23. Total Number of Patients/Subjects:
Group 1:	#Initially entered study________	Dropouts________	Final #_________
Group 2:	#Initially entered study________	Dropouts________	Final #_________
24. Sex (#):
Group 1:	Males________	Females________
Group 2:	Males________	Females________
25. Race/Ethnic Group:
26. Age (Y-M): Mean:________________	Median:___________________	SD:___________________	Range:___________________
27. Other Demographic Characteristics (Abstractor fill in as needed/appropriate)
28. Inclusion Criteria Reported (Highlight one):
1=Yes	2=No	3=Unclear
Page:_______________	Paragraph(s):_______________
(Abstractors: Note differences between groups 1 and 2, if any.)
1.______________________________________________________________________________________
2.______________________________________________________________________________________
3.______________________________________________________________________________________
4.______________________________________________________________________________________
29. Exclusion Criteria Reported (Highlight one):
1=Yes	2=No	3=Unclear
Page:_______________	Paragraph(s):_________
(Abstractors: Note differences between groups 1 and 2, if any.)
1.______________________________________________________________________________________
2.______________________________________________________________________________________
3.______________________________________________________________________________________
4.______________________________________________________________________________________
30. Intervention(s) Studied:
(Abstractor describe the therapy or intervention provided to study participants. If none given, write "none" on first line. Add or eliminate lines as needed to describe.)
Page:__________	Paragraph(s):_________
1.______________________________________________________________________________________
2.______________________________________________________________________________________
3.______________________________________________________________________________________
4.______________________________________________________________________________________
SECTION IV: Statistical Analysis
Statistical Methods Employed:
31. What statistics are employed? (Highlight all that apply)
1=Not reported		5=Regression modeling-linear
2=Unclear		6=Logistic regression modeling
3=t-tests, Z scores, ANOVA		6=Factor analysis
4=Correlations		7=Other (specify)________________________________
32. If different statistical methods are used to address the outcomes, describe which methods are used for each outcome:
Outcome #1: ________________________________
Statistical Method #1: _____________________________________________________________________
______________________________________________________________________________________
______________________________________________________________________________________
Outcome #2: ________________________________
Statistical Method #2: _____________________________________________________________________
______________________________________________________________________________________
______________________________________________________________________________________
Outcome #3: ________________________________
Statistical Method #3: _____________________________________________________________________
______________________________________________________________________________________
______________________________________________________________________________________
33. What factors were controlled for in the analyses? (e.g., age, gender, race, comorbidities, initial speech/language impairment, etc.)
1.______________________________________________________________________________________
2.______________________________________________________________________________________
3.______________________________________________________________________________________
4.______________________________________________________________________________________
SECTION V: Results
34. What were the outcome(s) observed for each group?
[Abstractor: Summarize results for each outcome, including statistic(s) used, and indicate source of data (page and paragraph number) for each group. Measures of central tendency or dispersion (i.e., standard deviation) and significance levels should be included, as appropriate.]
Outcome #1:________________________________
Page:_____________	Paragraph/Table:___________
______________________________________________________________________________________
______________________________________________________________________________________
______________________________________________________________________________________
______________________________________________________________________________________
Outcome #2:________________________________
Page:_______________	Paragraph/Table:____________
______________________________________________________________________________________
______________________________________________________________________________________
______________________________________________________________________________________
______________________________________________________________________________________
Outcome #3:__________________________
Page:_____________	Paragraph/Table:_____________
______________________________________________________________________________________
______________________________________________________________________________________
______________________________________________________________________________________
______________________________________________________________________________________
SECTION VI: Abstractor Notes and Comments
35. Limitations Noted in Article (provide page and paragraph numbers):
Page:___________	Paragraph(s):_________
______________________________________________________________________________________
______________________________________________________________________________________
______________________________________________________________________________________
______________________________________________________________________________________
______________________________________________________________________________________
______________________________________________________________________________________
______________________________________________________________________________________
______________________________________________________________________________________
36. Limitations Noted by Reviewer (provide page and paragraph numbers):
Page:_____________	Paragraph(s):__________
______________________________________________________________________________________
______________________________________________________________________________________
______________________________________________________________________________________
______________________________________________________________________________________
______________________________________________________________________________________
______________________________________________________________________________________
______________________________________________________________________________________
37. Abstractor Notes to Second Reviewer:
(Please highlight any areas or data in the article, listing page and paragraph(s), which do not fit into the abstraction form but you think are important for the second reviewer to read and evaluation.
Page:____________	Paragraph(s):________
Reason:________________________________________________________________________________
______________________________________________________________________________________
Page:___________	Paragraph(s):_______
Reason:________________________________________________________________________________
______________________________________________________________________________________
Page:_________	Paragraph(s):_____________
Reason:________________________________
______________________________________________________________________________________

Figure D3. Data Extraction Form for Instrument Manuals Criteria for Determining Disability in Speech Language Disorders (pages 315-334)

Appendix D: Methodology

DATA EXTRACTION FORM CRITERIA FOR DETERMINING DISABILITY IN SPEECH LANGUAGE DISORDERS

SECTION 1: Study Background Information

Date of Review (MM/DD/2001): ____ / ____ / 2001
Tool Used: (specify instrument(s)/version(s) tested first, criterion/reference test next)
Instrument #1:____________________________ Version: ___
Instrument #2:____________________________ Version: ___
Instrument #3: ____________________________ Version: ___
Instrument #4: ____________________________ Version: ___
Population (highlight one):	1= Children <=21 yrs	2=Adults <= 62 yrs
Disorder (highlight one):	1=Not reporte	2=Unclear	3=Language
	4=Speech	5=Voice
	6=Other (specify)_______________________________________________
Year Published (YYYY): __ __ __ __
Country where study conducted: __ __
1=United States	2=Canada	3=Britain/United Kingdom
4=Australia	5=New Zealand	6=European Country_______________
7=South America__________________		8=Asia__________________________
Number of Authors: __ __ (Put 1 if only testing service given)
Surname of First Author____________________________________
Background of First Author (highlight all that apply):
1=Not reported		8=Pediatrics
2=Unclear		9=Neurology
3=Speech Language Pathology		10=Otolaryngology
4=Audiology		11=Education
5=Psychology		12=Test agency/publisher
6=Neuropsychology		13=Other (specify)_______________
7=Psychiatry
Funding Source (highlight all that apply):
1=Not reported		5=Industry
2=Unclear		6=Government Agency
3=Consumer/Patient Organization		7=Professional Organization
4=Charity		8= Other (specify)__________________

SECTION II: Overall Study Description

Main Objectives (as described by author, give page and paragraph number in article):

Page:________

Paragraph(s):_________

________________________________________________________________________________________

__________________________________________________________________________________________

Study/Evaluation Setting (highlight all that apply):

1=Not reported

7=Speech/Language Clinic: Inpatient

2=Unclear

8=Speech/Language Clinic: Outpatient

3=Community

9=Hospital: Inpatient

4=School: Preschool

10=Hospital: Outpatient

5=School: Elementary School

11=Other (specify)__________________________

6=School: Middle/High School

Characteristics of Assessors/Evaluators

Number of Assessors/Evaluators (if given): __ __

Background of Assessors/Evaluators (highlight all that apply):

1=Not reported

7=Psychiatrist

2=Unclear

8=Pediatrician

3=Speech Language Pathologist

9=Neurologist

4=Audiologist

10=Otolaryngologist

5=Psychologist

11=Graduate Student: (specify discipline)

6=Neuropsychologist

→________________________

Characteristics of Patients/Subjects

Abstractor: Add or eliminate groups as required.

Group 1

Defining Characteristic(s)___________________________________________

Co-morbidities:____________________________________________________

Total Number of Patients/Subjects:

Group 1:	#Initially entered study_________		Dropouts_______	#_______________
Group 2:	#Initially entered study_________		Dropouts_______	#_______________
Sex (#):
Group 1:	Males_______	Females_______
Group 2:	Males_______	Females_______

Race/Ethnic Group:
Race/Ethnicity	Group 1		Group 2
Race/Ethnicity	#	%	#	%
Race/Ethnic Group:
Race/Ethnicity	Group 1		Group 2
Race/Ethnicity	#	%	#	%
Not Reported
Unclear
White
Black orAfrican American
Hispanic or Latino
Asian
American Indian or Alaskan Native
Native Hawaiian or Pacific Islander
Other (specify)

Age (Y-M):

Mean________

Median_______

SD___________

Range_____________

Age Group for Children: (Abstractor fill in as needed/appropriate)
Age Group (Y-M to Y-M)	Group 1		Group 2
Age Group (Y-M to Y-M)	#	%	#	%
Age Group for Children: (Abstractor fill in as needed/appropriate)
Age Group (Y-M to Y-M)	Group 1		Group 2
Age Group (Y-M to Y-M)	#	%	#	%

School Grade for Children: (Abstractor fill in as needed/appropriate)
Grades	Group 1		Group 2
Grades	#	%	#	%
School Grade for Children: (Abstractor fill in as needed/appropriate)
Grades	Group 1		Group 2
Grades	#	%	#	%
Pre-School

Inclusion Criteria Reported (Highlight one):
1=Yes	2=No	3=Unclear
Page:______	Paragraph(s):________
1.________________________________________________________________________________________
2.________________________________________________________________________________________
3.________________________________________________________________________________________
4.________________________________________________________________________________________

Exclusion Criteria Reported (Highlight one):
1=Yes	2=No	3=Unclear
Page:________	Paragraph(s):___________
1.________________________________________________________________________________________
2.________________________________________________________________________________________
3.________________________________________________________________________________________
4.________________________________________________________________________________________

Study Purpose (highlight all that apply):
Reliability Internal Consistency Test-Retest Inter-rater Intra-rater Other				Validity Content Construct Concurrent Divergent Predictive Other
Normative Data

SECTION III: Reliability Evaluations

A. Internal Consistency Reliability Evaluation Description

Description of Study Design (summarize and give page and paragraph number in article)

(Skip only if same as overall design description)

Page:_______

Paragraph(s):__________

__________________________________________________________________________________________

Dates of Study______________________________________________________________________________
(Provide month/year of starting and ending points, if given.)

Length of Study Follow-up____________________________________________________________________
(If follow-up times vary, include the range and mean, median, and/or mode if they are given.)

Study/Evaluation Setting (Note only differences from overall study):

__________________________________________________________________________________________

Characteristics of Assessors/Evaluators (Note only differences from overall study):

__________________________________________________________________________________________

Characteristics of Patients/Subjects (Note only differences from overall study):

__________________________________________________________________________________________

Statistical Methods Employed--Internal Consistency

What statistics are employed?(Highlight all that apply)

Page:______

Paragraph(s):_______

1=Not reported

4=Correlation Coefficient

2=Unclear

5=Item-total Correlation

3=Cronbach's Coefficient α

6=Other (specify)________________

Comments about how statistical methods are applied

__________________________________________________________________________________________

Results -- Internal Consistency Reliability

Abstractor: Summarize results, including statistic used, and indicate source of data (page and table number). Repeat section as necessary for subtests and overall score.

e.g., (Page 5, Table 2). Cronbach's alpha for sounds-in-words tests reported for each age group. Alphas range from 0.XX to 0.XX with a mean of 0.XX. Alphas reported separately for males (0.XX) and females (0.XX).....

Page:_________

Table(s):_____________

Range:________

Mean:__________

Median:________

__________________________________________________________________________________________

B. Test-Retest Reliability Evaluation;
Description of Study Design (summarize and give page and paragraph number in article)

(Skip only if same as overall design description)

Page:_________

Paragraph(s):________

__________________________________________________________________________________________

Dates of Study______________________________________________________________________________
(Provide month/year of starting and ending points, if given.) (Skip only if same as overall design description)

Study/Evaluation Setting (Note only differences from overall study):

__________________________________________________________________________________________

Characteristics of Assessors/Evaluators (Note only differences from overall study):

__________________________________________________________________________________________

Characteristics of Patients/Subjects (Note only differences from overall study):

__________________________________________________________________________________________

Statistical Methods Employed--Test-Retest Reliability

Is test-retest reliability reported for overall score?

1=Not reported

2=Unclear

3=No

4=Yes

What statistics are employed? (Highlight all that apply)

1=Not reported

4=Percent of agreement

2=Unclear

5=Correlation between scores

3=Cohen's Kappa

6=Other (specify)_________________

Is test-retest reliability reported on a per-item basis, if appropriate?

1=Not reported

2=Unclear

3=No

4=Yes

What statistics are employed? (Highlight all that apply)

1=Not reported

4=Percent of agreement

2=Unclear

5=Correlation between scores

3=Cohen's Kappa

6=Other (specify)_________________

Comments about how statistical methods are applied

__________________________________________________________________________________________

Results -- Test-Retest Reliability

Abstractor: Summarize results, including statistic used, and indicate source of data (page and table number). Repeat section as necessary for subtests and overall score.

e.g., (Page 5, Table 2). Cohen's Kappa for sounds-in-words tests reported for each age group for each of the three consonant
positions (initial/medial/final). Kappas for initial sounds range from 0.XX to 0.XX with a mean of 0.XX. Kappas for medial
sounds range from 0.XX to 0.XX with a mean of 0.XX. Kappas for final sounds range from range from 0.XX to 0.XX with a
mean of 0.XX...__________________________________________________________________________________

Page:____________

Table(s):__________

Range:___________

Mean:_______

Median:____________

__________________________________________________________________________________________

C. Inter-Rater Reliability Evaluation:

(Skip to next section only if same as overall design description)