Main Article Content
Abstract
The study aimed at investigating the reliability indices of the operational score calculated by three different methods of score resolution among evaluators' ratings of a writing task (the averaged score of the original ratings, the averaged score of the expert rating and the original ratings, the averaged score of the expert rating and the closest rating of the original ratings). Two scoring rubrics (five categories, seven categories) were used. The effect of score resolution method was examined, and scoring rubric in the operational score calculated in one of the above mentioned methods. Data were obtained from the answers of 232 male and female teachers to a previously developed writing task. The writings were blindly assessed by six raters using the two pre-developed scoring rubrics. The results revealed differences among reliability indices due to the resolution method used to calculate the operational score regardless of the scoring rubric being used. Repeated measures ANOVA with between-subjects factor revealed significant main effect of both factors i.e., score resolution method, and scoring rubric in the calculated operational score using one of the aforementioned resolution methods.