Just Read the Question: Enabling Generalization to New Assessment Items with Text Awareness

TL;DR

This paper introduces Text-LENS, a model that incorporates item text embeddings to improve generalization to new assessment items, demonstrating effectiveness on datasets with unseen items.

Contribution

The paper extends the LENS model by integrating text embeddings, enabling better generalization to new assessment items in educational settings.

Findings

Text-LENS matches LENS on seen items.

Text-LENS outperforms LENS on unseen items.

Effective learning of student proficiency from new items.

Abstract

Machine learning has been proposed as a way to improve educational assessment by making fine-grained predictions about student performance and learning relationships between items. One challenge with many machine learning approaches is incorporating new items, as these approaches rely heavily on historical data. We develop Text-LENS by extending the LENS partial variational auto-encoder for educational assessment to leverage item text embeddings, and explore the impact on predictive performance and generalization to previously unseen items. We examine performance on two datasets: Eedi, a publicly available dataset that includes item content, and LLM-Sim, a novel dataset with test items produced by an LLM. We find that Text-LENS matches LENS' performance on seen items and improves upon it in a variety of conditions involving unseen items; it effectively learns student proficiency from and makes predictions about student performance on new items.