Regularizing Recurrent Neural Networks via Sequence Mixup

TL;DR

This paper adapts mixup regularization techniques from feed-forward neural networks to RNNs, improving performance on sequence tasks with minimal added complexity.

Contribution

It introduces sequence mixup methods for RNNs, providing an easy-to-implement regularization approach validated through experiments and theoretical analysis.

Findings

Improved F-1 score on NER task

Reduced loss in RNN training

Validated with real-world datasets

Abstract

In this paper, we extend a class of celebrated regularization techniques originally proposed for feed-forward neural networks, namely Input Mixup (Zhang et al., 2017) and Manifold Mixup (Verma et al., 2018), to the realm of Recurrent Neural Networks (RNN). Our proposed methods are easy to implement and have a low computational complexity, while leverage the performance of simple neural architectures in a variety of tasks. We have validated our claims through several experiments on real-world datasets, and also provide an asymptotic theoretical analysis to further investigate the properties and potential impacts of our proposed techniques. Applying sequence mixup to BiLSTM-CRF model (Huang et al., 2015) to Named Entity Recognition task on CoNLL-2003 data (Sang and De Meulder, 2003) has improved the F-1 score on the test stage and reduced the loss, considerably.