Survey on Automated Short Answer Grading with Deep Learning: from Word Embeddings to Transformers

TL;DR

This survey reviews recent deep learning methods for automated short answer grading, emphasizing the transition from handcrafted features to representation learning, and highlights the effectiveness of combining both approaches.

Contribution

It provides a comprehensive analysis of recent deep learning techniques in ASAG, focusing on the shift from feature engineering to representation learning and their combined impact.

Findings

Deep learning methods improve ASAG performance when combined with hand-engineered features.

Transformers architectures provide the most semantic understanding for grading tasks.

Representation learning alone is less effective than when integrated with traditional features.

Abstract

Automated short answer grading (ASAG) has gained attention in education as a means to scale educational tasks to the growing number of students. Recent progress in Natural Language Processing and Machine Learning has largely influenced the field of ASAG, of which we survey the recent research advancements. We complement previous surveys by providing a comprehensive analysis of recently published methods that deploy deep learning approaches. In particular, we focus our analysis on the transition from hand engineered features to representation learning approaches, which learn representative features for the task at hand automatically from large corpora of data. We structure our analysis of deep learning methods along three categories: word embeddings, sequential models, and attention-based methods. Deep learning impacted ASAG differently than other fields of NLP, as we noticed that the learned representations alone do not contribute to achieve the best results, but they rather show to work in a complementary way with hand-engineered features. The best performance are indeed achieved by methods that combine the carefully hand-engineered features with the power of the semantic descriptions provided by the latest models, like transformers architectures. We identify challenges and provide an outlook on research direction that can be addressed in the future