Assessing the Reliability of a Chemical Engineering Problem-solving Rubric when Using Multiple Raters
Timothy Duckett; Matthew W. Liberatore; Uchenna Asogwa; Gale Mentzer; Amanda Malefyt; George Engelhard Jr.
This evidence-based practice paper discusses the preliminary validation of a project modified version of the Promoting Problem Solving Proficiency in First Year Engineering (PROCESS). The full rating plan required four raters to use the PROCESS to assess the problem-solving ability of ~70 engineering students randomly selected from two undergraduate cohorts at two Midwest universities. The many-facet Rasch measurement model has the psychometric properties to determine if there are any characteristics other than problem-solving that influence the scores assigned to students, such as rater bias or differential item functioning. Prior to implementing the full rating plan, the analysis examined how raters interacted with the six items on the modified PROCESS when scoring a random selection of 20 students' solutions to one textbook homework problem. Follow up inter-rater reliability meetings ...
Timothy Duckett, Matthew W. Liberatore, Uchenna Asogwa, Gale Mentzer, Amanda Malefyt, & George Engelhard Jr. (2020). Assessing the Reliability of a Chemical Engineering Problem-solving Rubric when Using Multiple Raters. 2019 ASEE Annual Conference & Exposition Proceedings. https://doi.org/10.18260/1-2--32124