Test-takers have a say: understanding the implications of the use of AI in language tests

TL;DR

This study explores how AI integration in language testing affects test-taker perceptions, revealing both positive and negative impacts on fairness, trust, and well-being, based on empirical data from interviews and surveys.

Contribution

First empirical investigation into test-taker perspectives on AI in language tests, highlighting perceptions and behavioral implications of AI adoption.

Findings

AI may improve perceptions of fairness and accessibility.

AI could cause mistrust in reliability and interactivity.

AI impacts test-taker well-being and behavior.

Abstract

Language tests measure a person's ability to use a language in terms of listening, speaking, reading, or writing. Such tests play an integral role in academic, professional, and immigration domains, with entities such as educational institutions, professional accreditation bodies, and governments using them to assess candidate language proficiency. Recent advances in Artificial Intelligence (AI) and the discipline of Natural Language Processing have prompted language test providers to explore AI's potential applicability within language testing, leading to transformative activity patterns surrounding language instruction and learning. However, with concerns over AI's trustworthiness, it is imperative to understand the implications of integrating AI into language testing. This knowledge will enable stakeholders to make well-informed decisions, thus safeguarding community well-being and testing integrity. To understand the concerns and effects of AI usage in language tests, we conducted interviews and surveys with English test-takers. To the best of our knowledge, this is the first empirical study aimed at identifying the implications of AI adoption in language tests from a test-taker perspective. Our study reveals test-taker perceptions and behavioral patterns. Specifically, we identify that AI integration may enhance perceptions of fairness, consistency, and availability. Conversely, it might incite mistrust regarding reliability and interactivity aspects, subsequently influencing the behaviors and well-being of test-takers. These insights provide a better understanding of potential societal implications and assist stakeholders in making informed decisions concerning AI usage in language testing.