Using Error/Tolerance Analysis to Design an Empirical Practice Analysis

Adv Health Sci Educ Theory Pract. 2000;5(3):179-196. doi: 10.1023/A:1009821413152.

Abstract

Practice analyses, in which professional practitioners describe their work, provide essential content-related validity evidence for licensure and certification tests by providing an empirical base for the test plan. This paper examines the precision of the category weights generated by practice analyses employing activity inventories. The standard errors in estimates of category weights are compared to the tolerance for error in these weights. The tolerance for error specifies how large errors can be before they interfere with the intended use of the measurement procedure. As an example, data from a practice-analysis study is used to assign weights to nine categories in a test plan. Estimates of standard errors are derived from G studies with activity statements nested within categories and crossed with respondents and occasions. The tolerance for error is specified in terms of the level of precision typically found in test plans. The results are used to determine the minimum number of activities needed for each category to satisfy the tolerance requirement.