Automated speaking assessment: What do measures of utterance fluency tell us about linguistic knowledge and processing?

Wednesday 19 June 2019, 12.00PM to 1.00pm

Speaker(s): Zoe Handley, Centre for Research in Language Learning and Use

As a result of recent advances in artificial intelligence and natural language processing automated language assessment is becoming a reality and there is growing interest in the development automated speaking assessments. Evidence for their validity of the scores generated by such systems is also growing, including evidence of concurrent validity with other measures of oral proficiency, their capacity to discriminate between learners at different proficiency levels, and their capacity to detect improvements in oral proficiency following an intervention. Measures of utterance fluency typically form the basis of the scoring models that underpin these assessments. Little is however known about what precisely these measures, and hence these tests, tell us about learners’ underlying linguistic knowledge and processing.

This paper explores this question through the analysis of a data set collected from seventy-three advanced Chinese learners of English. Each learner completed seven tasks: 1) an IELTS-style monologic narrative speaking task, 2) the productive levels test, 3) the word associates test, 4) a picture naming task, 5) a grammar knowledge test, 6) a sentence inflection and agreement task designed to measure morpho-syntactic encoding task, and 7) a sentence transformation task designed to measure syntactic encoding.

The most frequently employed measures of utterance fluency employed in automated systems were calculated by hand based on the learners’ performance in the IELTS style monologic speaking task. Accuracy and reaction times were recorded for the picture naming and sentence construction tasks. Correlation and regression analyses were then conducted to examine the relationship between the different measures of utterance fluency and linguistic knowledge and processing. Implications of the findings for automated speaking assessment with a focus on implications for the interpretation of the scores generated by such tests will be discussed.

Location: H/G21, Heslington Hall