extended · article · 1984

ESTIMATING CRITERION-REFERENCED STANDARDS FOR MULTIPLE-CHOICE EXAMINATIONS

Francis P. Hughes; Charles F. Schumacher; Benjamin D. Wright

Four methods were studied for setting a standard on a written examination containing several clinical discipline subtests. The NBME method yielded the most consistent estimate. The Angoff and Ebel methods yielded slightly less consistent estimates. All estimates were more consistent when computed as the average of discipline standards rather than the judges' personal standards. All but the Essential Content method yielded similar and practical estimates of the standard. The Ebel method was not found to be feasible for use with the examination studied; however, incorporating clusters of equally difficult, relevant items and feedback based on the judges' previous judgments into the NBME method yielded subsequent estimates of the standard that were more consistent than, but not significantly different from, the first estimate. The findings suggest the importance of enabling judges to select...