UmBERTo-MTSA @ AcCompl-It: Improving Complexity and Acceptability Prediction with Multi-task Learning on Self-Supervised Annotations

TL;DR

This paper presents a self-supervised data augmentation method using multi-task learning on annotations generated by multiple model copies, significantly enhancing neural language models' performance in complexity and acceptability prediction tasks.

Contribution

It introduces a novel self-supervised augmentation technique that leverages multi-task learning on pseudo-annotated data to improve model accuracy with limited labeled data.

Findings

Significant improvement in prediction quality for complexity and acceptability tasks.

Effective use of unlabeled data through self-supervised annotation.

Enhanced model robustness via multi-task training on pseudo-labels.

Abstract

This work describes a self-supervised data augmentation approach used to improve learning models' performances when only a moderate amount of labeled data is available. Multiple copies of the original model are initially trained on the downstream task. Their predictions are then used to annotate a large set of unlabeled examples. Finally, multi-task training is performed on the parallel annotations of the resulting training set, and final scores are obtained by averaging annotator-specific head predictions. Neural language models are fine-tuned using this procedure in the context of the AcCompl-it shared task at EVALITA 2020, obtaining considerable improvements in prediction quality.