Out-of-Manifold Regularization in Contextual Embedding Space for Text Classification

TL;DR

This paper introduces a novel regularization method for text classification that leverages out-of-manifold embeddings, generated and discriminated through a joint model, improving model robustness and performance.

Contribution

It proposes a new out-of-manifold regularization technique using a generator-discriminator framework to enhance neural network training in text classification.

Findings

Improves classification accuracy on multiple benchmarks.

Compatible with existing data augmentation methods.

Effectively regularizes the embedding space outside the manifold.

Abstract

Recent studies on neural networks with pre-trained weights (i.e., BERT) have mainly focused on a low-dimensional subspace, where the embedding vectors computed from input words (or their contexts) are located. In this work, we propose a new approach to finding and regularizing the remainder of the space, referred to as out-of-manifold, which cannot be accessed through the words. Specifically, we synthesize the out-of-manifold embeddings based on two embeddings obtained from actually-observed words, to utilize them for fine-tuning the network. A discriminator is trained to detect whether an input embedding is located inside the manifold or not, and simultaneously, a generator is optimized to produce new embeddings that can be easily identified as out-of-manifold by the discriminator. These two modules successfully collaborate in a unified and end-to-end manner for regularizing the out-of-manifold. Our extensive evaluation on various text classification benchmarks demonstrates the effectiveness of our approach, as well as its good compatibility with existing data augmentation techniques which aim to enhance the manifold.