SYSTEMATIC RESEARCH SYNTHESIS ON RATING IN ESP SPEAKING ASSESSMENTS

Yuko Hijikata, Jiyu Min

DOI Number
https://doi.org/10.22190/JTESAP2103465H
First page
465
Last page
473

Abstract


Jacoby and McNamara (1998) insist that domain specialists (e.g., doctors) and language specialists (e.g., teachers) differ in their approaches to evaluating profession-specific communication tasks. However, due to the scarcity of studies examining speaking assessments for occupational purposes, the differences between the two rater groups have not yet been clearly revealed. In response to this gap in the literature, this systematic research synthesis study examines how the workplace speaking skill has been evaluated, focusing on rater groups and rating scales. The major findings are as follows. First, research on ESP speaking assessments tends to include more than one rater group such as domain specialists and language specialists. Second, while domain specialists and language specialists generally demonstrate high intergroup correlations, the rater group notably differs in terms of field-specific criteria. Third, compared with linguistic scales, field-specific criteria have not been developed. Based on these results, directions for future research are discussed.

Keywords

English for specific purposes, English for occupational purposes, speaking, assessment, rating, systematic research synthesis

Full Text:

PDF

References


Brown, Anne. 1995. The effect of rater variables in the development of an occupation-specific language performance test. Language Testing, 12: 1–16. doi:10.1177/026553229501200101

Cowling, Jeremy. 2007. Needs analysis: Planning a syllabus for a series of intensive workplace courses at a leading Japanese company. English for Specific Purposes 26:426–442. doi:10.1016/j.esp.2006.10.003

Douglas, Dan. 2000. Assessing Language for Specific Purposes. Cambridge: Cambridge University Press.

Douglas, Dan. 2001. Language for specific purposes assessment criteria: Where do they come from? Language Testing 18:171–185. doi:10.1177/026553220101800204

Douglas, Dan. 2013. ESP and assessment. In The Handbook of English for Specific Purposes, edited by B. Paltridge and S. Starfield, 367–383. New York: John Wiley & Sons.

Duijm, Klaartje, Rob Schoonen, and Jan Hulstijn. 2018. Professional and non-professional raters’ responsiveness to fluency and accuracy in L2 speech: An experimental approach. Language Testing 35:501–527. doi:10.1177/0265532217712553

Ekkens, Kristin, and Paula Winke. 2009. Evaluating workplace English language programs. Language Assessment Quarterly 6:265–287. doi:10.1080/15434300903063038

Elder, Catherine, and Tim McNamara. 2016. The hunt for “indigenous criteria” in assessing communication in the physiotherapy workplace. Language Testing 33:153–174. doi:10.1177/0265532215607398

Elder, Catherine. 1993. How do subject specialists construe classroom language proficiency? Language Testing 10:235–254. doi:10.1177/026553229301000303

Friginal, Eric. 2013. Evaluation of oral performance in outsourced call centres: An exploratory case study. English for Specific Purposes 32:25–35. doi:10.1016/j.esp.2012.06.002

Han, Chao. 2016. Investigating score dependability in English/Chinese interpreter certification performance testing: A generalizability theory approach. Language Assessment Quarterly 13:186–201. doi:10.1080/15434303.2016.1211132

Hashemi, Mohammad, and Esmat Babaii. 2013. Mixed methods research: Toward new research designs in applied linguistics. Modern Language Journal 97:828–852. doi:10.1111/j.1540-4781.2013.12049.x

In’nami, Yo, and Rie Koizumi. 2009. A meta-analysis of test format effects on reading and listening test performance: Focus on multiple-choice and open-ended formats. Language Testing 26:219–244. doi:10.1177/0265532208101006

In’nami, Yo, and Rie Koizumi. 2016. Task and rater effects in L2 speaking and writing: A synthesis of generalizability. Language Testing 33: 341–366. doi:10.1177/0265532215587390

Jacoby, Sally, and Tim McNamara. 1999. Locating competence. English for Specific Purposes 18:213–241. doi:10.1016/S0889-4906(97)00053-7

Kim, Hyejeong. 2018. What constitutes professional communication in aviation: Is language proficiency enough for testing purposes? Language Testing 35:403–426. doi:10.1177/0265532218758127

Kim, Hyejeong, and Rosey Billington. 2016. Pronunciation and comprehension in English as a lingua franca communication: Effect of L1 influence in international aviation communication. Applied Linguistics 39:135–158. doi:10.1093/applin/amv075

Knoch, Ute. 2014. Using subject specialists to validate an ESP rating scale: The case of the International Civil Aviation Organization (ICAO) rating scale. English for Specific Purposes 33:77–86. doi:10.1016/j.esp.2013.08.002

Knoch, Ute, and Susy Macqueen. 2020. Assessing English for Professional Purposes. New York: Routledge.

Lumley, Tom. 1998. Perceptions of language-trained raters and occupational experts in a test of occupational English language proficiency. English for Specific Purposes 17:347–367. doi:10.1016/S0889-4906(97)00016-1

Manias, Elizabeth, and Tim McNamara. 2016. Standard setting in specific-purpose language testing: What can a qualitative study add? Language Testing 33:235–249. doi:10.1177/0265532215608411

O’Hagan, Sally, John Pill, and Ying Zhang. 2016. Extending the scope of speaking assessment criteria in a specific-purpose language test: Operationalizing a health professional perspective. Language Testing 33:195–216. doi:10.1177/0265532215607920

Pill, John, and Tim McNamara. 2016. How much is enough? Involving occupational experts in setting standards on a specific-purpose language test for health professionals. Language Testing 33:217–234. doi:10.1177/0265532215607402

van Naerssen, Margaret, and Heidi Riggenbach. 1987. Evaluating oral skills in an EST program: Native English-speaking scientists respond to Chinese scientists' English. English for Specific Purposes 6:219–232. doi:10.1016/0889-4906(87)90005-6

Wette, Rosemary, and Susan Hawken. 2016. Measuring gains in an EMP course and the perspectives of language and medical educators as assessors. English for Specific Purposes 42:38–49. doi:10.1016/j.esp.2015.11.002




DOI: https://doi.org/10.22190/JTESAP2103465H

Refbacks

  • There are currently no refbacks.


ISSN 2334-9182 (Print)
ISSN 2334-9212 (Online)