Self-evaluation of rater bias in written composition assessment

• Published in: Linguistica
• Main Author: Sokolov, Cvetka
• Format: Article in Journal/Newspaper
• Language: English
• Published: Znanstvena založba Filozofske fakulte / Ljubljana University Press, Faculty of Arts 2014
• Subjects: rater bias; types of rater bias; self-evaluation; increased awareness; reliable and valid assessment; pristranskost ocenjevalcev; vrste pristranskosti; samoocenjevanje; višja stopnja ozaveščenosti; zanesljivo in veljavno ocenjevanje; sami
• Online Access: https://revije.ff.uni-lj.si/linguistica/article/view/2603; https://doi.org/10.4312/linguistica.54.1.261-275

No assessment is entirely free of bias. This paper presents findings concerning the way raters in the research group evaluate the extent to which they are influenced by various types of rater bias when grading their students’ written compositions. The sources of bias covered in the article include the teacher’s knowing the student writer and his or her proficiency in English, the difficulty of the writing task, distressful content likely to trigger the rater’s emotional reaction, the test taker’s views clashing with those of the rater, students’ progress, and the like. The data were gathered by the participants in the study via a questionnaire. In addition, the researcher’s interpretation of the respondents’ answers was verified through interviews. Although the two research methods and self-evaluation have their drawbacks, the results reveal interesting, relevant and important information on aspects which make written composition assessment less reliable and valid. The findings confirm the need to raise raters’ awareness of the causes of bias to which they are most susceptible, bringing them closer to effectively addressing the problem of assessment bias. The research involving eleven lecturers teaching Language in Use at the Department of English and American Studies at the Faculty of Arts, University of Ljubljana, is a part of a much larger project based on the author’s PhD thesis. Vsako ocenjevanje je do neke mere subjektivno. Članek predstavlja izsledke raziskave, ki išče odgovor na vprašanje, pod kakšnim vplivom različnih oblik pristranskosti so po lastni presoji ocenjevalci in ocenjevalke, ki so v raziskavi sodelovali. Do kakšne mere na končno oceno vplivajo dejavniki, kot na primer težavnost naslova/naloge, pretresljive in osebne vsebine v sestavku, ki utegnejo pri ocenjevalcu/ocenjevalki sprožiti čustven odziv, avtorjevo/avtoričino stališče, s katerim se ocenjevalec/ocenjevalka ne strinja, študentov/študentkin napredek in dejstvo, da ocenjevalci/ocenjevalke avtorja/avtorico besedila poznajo in da vedo, kako dobro obvlada angleščino? Podatke smo zbrali s pomočjo vprašalnika, interpretacijo odgovorov udeležencev v raziskavi pa smo preverili z intervjuji. Čeprav imajo tako obe raziskovalni metodi kot samoocenjevanje določene pomanjkljivosti, prinašajo rezultati raziskave zanimiva, relevantna in pomembna spoznanja o vidikih ocenjevanja, ki zmanjšujejo njegovo zanesljivost in veljavnost. Raziskava potrjuje potrebo po ozaveščanju ocenjevalcev in ocenjevalk o različnih oblikah pristranskosti in jih spodbuja k razmisleku, za katere vplive so sami posebej dovzetni. Seznanjenost z možnimi viri pristranskosti in kritičen pretres lastnih ravnanj/odločitev v procesu ocenjevanja učitelje in učiteljice približata cilju – čim bolj objektivnemu ocenjevanju. Raziskava zajema 9 lektoric in 2 lektorja, ki na Oddelku za anglistiko in amerikanistiko Filozofske fakultete Univerze v Ljubljani učijo predmet Jezik v rabi. Gre za del mnogo obsežnejše raziskave, obdelane v avtoričini doktorski disertaciji.