extended · article · 2026

Revisiting reliability with human and machine learning raters under scoring design and rater configuration in the many‐facet Rasch model

Xingyao Xiao; Richard J. Patz; Mark R. Wilson; Mark Wilson

<jats:title>Abstract</jats:title> Constructed‐response (CR) items are widely used to assess higher order skills but require human scoring, which introduces variability and is costly at scale. Machine learning (ML)‐based scoring offers a scalable alternative, yet its psychometric consequences in rater‐mediated models remain underexplored. This study examines how scoring design, rater bias, ML inconsistency and model specification affect the reliability of ability estimation in polytomous CR assessments. Using Monte Carlo simulation, we manipulated human and ML rater bias, ML inconsistency and scoring density (complete, overlapping, isolated). Five estimation models were compared, including the Partial Credit Model (PCM) with fixed thresholds and the Many‐Facet Partial Credit Model (MFPCM) with and without free calibration. Results showed that systematic bias, not random ...