Joint Multi-Domain Learning for Automatic Short Answer Grading

TL;DR

This paper introduces JMD-ASAG, a novel deep learning model that effectively grades short answers across multiple domains by learning both generic and domain-specific features, outperforming existing methods.

Contribution

The paper presents the first joint multi-domain deep learning architecture for automatic short answer grading that adapts to multiple domains with limited data and surpasses prior models.

Findings

JMD-ASAG outperforms existing techniques on large-scale industry and benchmark datasets.

The model achieves state-of-the-art results on the benchmark dataset.

It effectively learns domain-specific and generic features for answer grading.

Abstract

One of the fundamental challenges towards building any intelligent tutoring system is its ability to automatically grade short student answers. A typical automatic short answer grading system (ASAG) grades student answers across multiple domains (or subjects). Grading student answers requires building a supervised machine learning model that evaluates the similarity of the student answer with the reference answer(s). We observe that unlike typical textual similarity or entailment tasks, the notion of similarity is not universal here. On one hand, para-phrasal constructs of the language can indicate similarity independent of the domain. On the other hand, two words, or phrases, that are not strict synonyms of each other, might mean the same in certain domains. Building on this observation, we propose JMD-ASAG, the first joint multidomain deep learning architecture for automatic short answer grading that performs domain adaptation by learning generic and domain-specific aspects from the limited domain-wise training data. JMD-ASAG not only learns the domain-specific characteristics but also overcomes the dependence on a large corpus by learning the generic characteristics from the task-specific data itself. On a large-scale industry dataset and a benchmarking dataset, we show that our model performs significantly better than existing techniques which either learn domain-specific models or adapt a generic similarity scoring model from a large corpus. Further, on the benchmarking dataset, we report state-of-the-art results against all existing non-neural and neural models.