A pilot study of the effects of background characteristics of interviewers on the inter-rater reliability of the oral testing procedure for the senior high school French program in the province of Newfoundland

• Main Author: Flynn, Kevin Francis
• Other Authors: Memorial University of Newfoundland. Faculty of Education
• Format: Thesis
• Language: English
• Published: 1991
• Subjects: French language > Study and teaching (Secondary) > Newfoundland and Labrador > Evaluation; French language > Ability testing; Newfoundland; Canada; Newfoundland studies; University of Newfoundland
• Online Access: http://collections.mun.ca/cdm/ref/collection/theses2/id/263353

Thesis (M.Ed.)--Memorial University of Newfoundland, 1991. Education Bibliography: leaves 122-132. Although numerous studies have shown a high degree of validity and reliability in oral proficiency testing, not all areas have been investigated. This paper is, essentially, the report of a pilot study investigating one such area: the relationship between background characteristics of teacher interviewers and the reliability of the rating of an oral proficiency test. It was hypothesized that, despite the formal training sessions offered to the interviewers, differences in the ratings of the oral proficiency of interviewers would occur, and that these differences could be related to the diverse characteristics of the teacher interviewer. The questions asked were: -- (1) Are there significant differences in the way interviewers -- (a) rank the five factors of speaking proficiency? -- (b) rate the acceptability of errors? -- (c) rate proficiency levels? -- (2) If significant variations do occur, could these differences be associated with the language proficiency or the background characteristics of the interviewers? -- A questionnaire was distributed to one-half of the trained interview population on the island portion of the province. A response rate of 84 percent was achieved. The background characteristics investigated were: -- (1) demographic information, -- (2) number of years of teaching experience, -- (3) number of university French courses taken, -- (4) time spent in a French milieu, -- (5) extent of speaking French in the milieu with friends and acquaintances and at home, -- (6) a self-rating of aural and oral proficiency levels. The dependent variables employed were: -- (1) a ranking of the importance of five factors (vocabulary, grammar, pronunciation, fluency and comprehension) of oral proficiency; -- (2) a rating of the acceptability of examples of various types of errors, and -- (3) a rating of described proficiency levels as employed in the oral testing manual for the French 3200 Senior High School program. -- The data were tabulated and a profile of the average respondent constructed; different variables were analyzed to determine significant differences, then crosstabulated with the independent variable and a chi test performed. -- The results indicated that: -- (1) there were differences in the rankings of the five factors of oral proficiency, -- (2) there were significant differences in the rating of the acceptability of errors and the levels of oral proficiency, -- (3) these differences were primarily in the areas of the rating of vocabulary and grammar examples, -- (4) these differences were associated primarily with the language proficiency of the interviewers. -- Further study of this area is recommended in order to determine to what extent differences would affect student evaluation when using global rather than discrete item evaluation techniques.