Impact of item-writing flaws in multiple-choice questions on student achievement in high-stakes nursing assessments

Med Educ. 2008 Feb;42(2):198-206. doi: 10.1111/j.1365-2923.2007.02957.x.

Abstract

Context: Multiple-choice questions (MCQs) are frequently used to assess students in health science disciplines. However, few educators have formal instruction in writing MCQs and MCQ items often have item-writing flaws. The purpose of this study was to examine the impact of item-writing flaws on student achievement in high-stakes assessments in a nursing programme in an English-language university in Hong Kong.

Methods: From a larger sample, we selected 10 summative test papers that were administered to undergraduate nursing students in 1 nursing department. All test items were reviewed for item-writing flaws by a 4-person consensus panel. Items were classified as 'flawed' if they contained > or = 1 flaw. Items not containing item-writing violations were classified as 'standard'. For each paper, 2 separate scales were computed: a total scale which reflected the characteristics of the assessment as administered and a standard scale which reflected the characteristics of a hypothetical assessment including only unflawed items.

Results: The proportion of flawed items on the 10 test papers ranged from 28-75%; 47.3% of all items were flawed. Fewer examinees passed the standard scale than the total scale (748 [90.6%] versus 779 [94.3%]). Conversely, the proportion of examinees obtaining a score > or = 80% was higher on the standard scale than the total scale (173 [20.9%] versus 120 [14.5%]).

Conclusions: Flawed MCQ items were common in high-stakes nursing assessments but did not disadvantage borderline students, as has been previously demonstrated. Conversely, high-achieving students were more likely than borderline students to be penalised by flawed items.

Publication types

  • Research Support, Non-U.S. Gov't

MeSH terms

  • Choice Behavior
  • Education, Nursing / methods*
  • Educational Measurement / methods*
  • Educational Status
  • Hong Kong
  • Teaching / methods*