extended · article · 1998

Many-Faceted Rasch Modeling Expert Judgment in Test Development

Weimo Zhu; Catherine D. Ennis; Ang Chen; Benjamin D. Wright

The purpose of this study was to model expert judgment in test and instrument development using the manyfaceted Rasch model. A 150-item value orientation inventory-2 (VOI-2) assessing the value of physical education curriculum goals was developed and evaluated by 128 university educators and 103 school-based physical educators. The experts were asked to rate the consistency of each item to represent one part of the broad curriculum goals using a 5-point rating scale. The many-faceted Rasch model was used to calibrate the rating scores, and 6 facets-gender, ethnicity, employment type, rater, content area, and item-were defined. Severity and consistency of the experts' judgments were examined and corrected before being applied to item evaluation. Further, the impact of group membership on expert judgment was examined. Items were then evaluated based on their logit scores and the consistenc...