Automatically assessing oral narratives of Afrikaans and isiXhosa children

TL;DR

This paper introduces an automated system that assesses oral narratives of preschool children in Afrikaans and isiXhosa, aiding early literacy development by identifying children needing intervention.

Contribution

It presents a novel automated assessment system using speech recognition and machine learning, comparing linear models with large language models for scoring accuracy.

Findings

LLM-based system outperforms linear models in most cases

System is comparable to human experts in identifying children needing intervention

Provides a foundation for automatic oral assessments in classrooms

Abstract

Developing narrative and comprehension skills in early childhood is critical for later literacy. However, teachers in large preschool classrooms struggle to accurately identify students who require intervention. We present a system for automatically assessing oral narratives of preschool children in Afrikaans and isiXhosa. The system uses automatic speech recognition followed by a machine learning scoring model to predict narrative and comprehension scores. For scoring predicted transcripts, we compare a linear model to a large language model (LLM). The LLM-based system outperforms the linear model in most cases, but the linear system is competitive despite its simplicity. The LLM-based system is comparable to a human expert in flagging children who require intervention. We lay the foundation for automatic oral assessments in classrooms, giving teachers extra capacity to focus on personalised support for children's learning.