extended · book_chapter · 2007

Exploring rater judging behaviour using the many-facet rasch model

Noor Lide Abu Kassim; Noor Lide; George Engelhard Jr.

Performance assessment, unlike the traditional fixed-response assessment, has features peculiar to its assessment setting (the task choice, the task processing conditions, the raters,the rating scale, and the rating procedures) that make it much more vulnerable to construct irrelevant variance (McNamara, 1997; Upshur & Turner, 1999). Of these potential sources of variability, those associated with raters are considered to be extensive and pose serious threats to the validity of ratings (Linacre, 1989; McNamara, 1996).For performance assessment to yield valid and reliable results, it is essential that these sources of variability are eliminated or minimized. This paper illustrates how sources of rater-related variability or rater effects can be identified and controlled for using the Many-facet Rasch Model.It also illustrates how the idiosyncratic rating behaviour of individual raters can...