Should candidate scores be adjusted for interviewer stringency or leniency in the multiple mini-interview?

Med Educ. 2010 Jul;44(7):690-8. doi: 10.1111/j.1365-2923.2010.03689.x.

Abstract

Context: There are significant levels of variation in candidate multiple mini-interview (MMI) scores caused by interviewer-related factors. Multi-facet Rasch modelling (MFRM) has the capability to both identify these sources of error and partially adjust for them within a measurement model that may be fairer to the candidate.

Methods: Using facets software, a variance components analysis estimated sources of measurement error that were comparable with those produced by generalisability theory. Fair average scores for the effects of the stringency/leniency of interviewers and question difficulty were calculated and adjusted rankings of candidates were modelled.

Results: The decisions of 207 interviewers had an acceptable fit to the MFRM model. For one candidate assessed by one interviewer on one MMI question, 19.1% of the variance reflected candidate ability, 8.9% reflected interviewer stringency/leniency, 5.1% reflected interviewer question-specific stringency/leniency and 2.6% reflected question difficulty. If adjustments were made to candidates' raw scores for interviewer stringency/leniency and question difficulty, 11.5% of candidates would see a significant change in their ranking for selection into the programme. Greater interviewer leniency was associated with the number of candidates interviewed.

Conclusions: Interviewers differ in their degree of stringency/leniency and this appears to be a stable characteristic. The MFRM provides a recommendable way of giving a candidate score which adjusts for the stringency/leniency of whichever interviewers the candidate sees and the difficulty of the questions the candidate is asked.

MeSH terms

  • Clinical Competence
  • Communication
  • Educational Measurement / methods*
  • Educational Measurement / standards
  • Faculty, Medical
  • Humans
  • Interviews as Topic*
  • Observer Variation
  • Psychometrics / methods
  • School Admission Criteria*