Generalizable and Efficient Automated Scoring with a Knowledge-Distilled Multi-Task Mixture-of-Experts

TL;DR

This paper introduces UniMoE-Guided, a knowledge-distilled multi-task Mixture-of-Experts model that efficiently combines multiple task-specific models into a single, scalable system for automated scoring, maintaining high performance while reducing resource requirements.

Contribution

It presents a novel multi-task MoE approach that distills knowledge from large models into a compact, efficient student model capable of handling multiple scoring tasks.

Findings

Achieves performance comparable to task-specific models.

Uses approximately 6 times less storage than separate models.

Reduces storage needs by 87 times compared to large teacher models.

Abstract

Automated scoring of written constructed responses typically relies on separate models per task, straining computational resources, storage, and maintenance in real-world education settings. We propose UniMoE-Guided, a knowledge-distilled multi-task Mixture-of-Experts (MoE) approach that transfers expertise from multiple task-specific large models (teachers) into a single compact, deployable model (student). The student combines (i) a shared encoder for cross-task representations, (ii) a gated MoE block that balances shared and task-specific processing, and (iii) lightweight task heads. Trained with both ground-truth labels and teacher guidance, the student matches strong task-specific models while being far more efficient to train, store, and deploy. Beyond efficiency, the MoE layer improves transfer and generalization: experts develop reusable skills that boost cross-task performance and enable rapid adaptation to new tasks with minimal additions and tuning. On nine NGSS-aligned science-reasoning tasks (seven for training/evaluation and two held out for adaptation), UniMoE-Guided attains performance comparable to per-task models while using $\sim$6$\times$ less storage than maintaining separate students, and $87\times$ less than the 20B-parameter teacher. The method offers a practical path toward scalable, reliable, and resource-efficient automated scoring for classroom and large-scale assessment systems.