Small, Private Language Models as Teammates for Educational Assessment Design

TL;DR

This study compares small and large language models for educational assessment question design, highlighting SLMs' privacy advantages and their competitive quality, while noting evaluation inconsistencies.

Contribution

It systematically evaluates SLMs versus LLMs for assessment tasks, focusing on quality, reliability, and deployment considerations in educational settings.

Findings

SLMs perform competitively across pedagogical quality metrics.

Model-based evaluations show systematic biases compared to experts.

SLMs enable privacy-sensitive, local deployment for assessment design.

Abstract

Generative AI increasingly supports educational design tasks, e.g., through Large Language Models (LLMs), demonstrating the capability to design assessment questions that are aligned with pedagogical frameworks (e.g., Bloom's taxonomy). However, they often rely on subjective or limited evaluation methods; focus primarily on proprietary models; or rarely systematically examine generation, evaluation, or deployment constraints in real educational settings. Meanwhile, Small Language Models (SLMs) have emerged as local alternatives that better address privacy and resource limitations; yet their effectiveness for assessment tasks remains underexplored. To address this gap, we systematically compare LLMs and SLMs for assessment question design; evaluate generation quality across Bloom's taxonomy levels using reproducible, pedagogically grounded metrics; and further assess model-based judging against expert-informed evaluation by analyzing reliability and agreement patterns. Results show that SLMs achieve competitive performance across key pedagogically motivated quality dimensions while enabling local, privacy-sensitive deployment. However, model-based evaluations also exhibit systematic inconsistencies and bias relative to expert ratings. These findings provide evidence to posit language models as bounded assistants in assessment workflows; underscore the necessity of Human-in-the-Loop; and advance the automated educational question generation field by examining quality, reliability, and deployment-aware trade-offs.