Zero-Shot Grammar Competency Estimation Using Large Language Model Generated Pseudo Labels

TL;DR

This paper introduces a zero-shot framework for estimating grammar proficiency using large language models to generate pseudo labels from unlabeled data, reducing the need for manual annotation and enabling scalable assessment.

Contribution

It presents a novel training method that leverages LLM-generated pseudo labels for grammar scoring without manual labels, improving scalability and robustness.

Findings

Model performance depends on the choice of LLM for pseudo-labeling.

The ratio of clean to noisy samples affects training stability and accuracy.

The approach demonstrates high accuracy in grammar competency estimation.

Abstract

Grammar competency estimation is essential for assessing linguistic proficiency in both written and spoken language; however, the spoken modality presents additional challenges due to its spontaneous, unstructured, and disfluent nature. Developing accurate grammar scoring models further requires extensive expert annotation, making large-scale data creation impractical. To address these limitations, we propose a zero-shot grammar competency estimation framework that leverages unlabeled data and Large Language Models (LLMs) without relying on manual labels. During training, we employ LLM-generated predictions on unlabeled data by using grammar competency rubric-based prompts. These predictions, treated as pseudo labels, are utilized to train a transformer-based model through a novel training framework designed to handle label noise effectively. We show that the choice of LLM for pseudo-label generation critically affects model performance and that the ratio of clean-to-noisy samples during training strongly influences stability and accuracy. Finally, a qualitative analysis of error intensity and score prediction confirms the robustness and interpretability of our approach. Experimental results demonstrate the efficacy of our approach in estimating grammar competency scores with high accuracy, paving the way for scalable, low-resource grammar assessment systems.