Handwriting recognition and automatic scoring for descriptive answers in Japanese language tests

TL;DR

This paper develops a deep learning-based system for automatically recognizing and scoring handwritten Japanese descriptive answers in large-scale university entrance exams, achieving high accuracy and scoring agreement with human examiners.

Contribution

It introduces a novel approach combining ensemble neural recognizers and language models to effectively recognize unlabelled handwriting and automatically score descriptive answers.

Findings

Character recognition accuracy exceeds 97% on a small labeled dataset.

Automatic scoring achieves a Quadratic Weighted Kappa of 0.84 to 0.98.

Method demonstrates promising potential for end-to-end scoring of handwritten descriptive responses.

Abstract

This paper presents an experiment of automatically scoring handwritten descriptive answers in the trial tests for the new Japanese university entrance examination, which were made for about 120,000 examinees in 2017 and 2018. There are about 400,000 answers with more than 20 million characters. Although all answers have been scored by human examiners, handwritten characters are not labeled. We present our attempt to adapt deep neural network-based handwriting recognizers trained on a labeled handwriting dataset into this unlabeled answer set. Our proposed method combines different training strategies, ensembles multiple recognizers, and uses a language model built from a large general corpus to avoid overfitting into specific data. In our experiment, the proposed method records character accuracy of over 97% using about 2,000 verified labeled answers that account for less than 0.5% of the dataset. Then, the recognized answers are fed into a pre-trained automatic scoring system based on the BERT model without correcting misrecognized characters and providing rubric annotations. The automatic scoring system achieves from 0.84 to 0.98 of Quadratic Weighted Kappa (QWK). As QWK is over 0.8, it represents an acceptable similarity of scoring between the automatic scoring system and the human examiners. These results are promising for further research on end-to-end automatic scoring of descriptive answers.