extended · article · 2018

A Study on Rater Reliability Under Holistic and Analytic Scoring of CEPT Writing by Using Generalizability Theory and Many-facet Rasch Model

Lin Chun; Xiao Yun-nan; George Engelhard Jr.

To explore the scoring reliability of College English Placement Test (CEPT) writing, generalizability theory (GT) and many-facet Rasch model (MFRM) were applied to analyze 15 raters' holistic and analytic ratings of 300 writing samples. The results were as follows: (1) Raters' scoring method had significant impact on their rating of CEPT writing; under either method, one rater was enough to ensure that the generalizability coefficient was 0.8 or above. (2) Whichever method was adopted, raters differed significantly from each other in severity, but they had sound intra-rater consistency; raters were most biased towards task, then grammar, mechanics, vocabulary, and were least biased towards structure; if the dimensions at the discourse level were scored severely, then those at the linguistic level were scored leniently, and vice versa; raters tended to be severely biased towards the low, ...