ARCH: Efficient Adversarial Regularized Training with Caching

TL;DR

ARCH introduces a caching-based adversarial regularization method that reduces computational costs and improves model generalization in NLP tasks by caching perturbations and using a K-nearest neighbors strategy.

Contribution

The paper presents ARCH, a novel adversarial regularization approach that caches perturbations to save computation and employs a K-nearest neighbors strategy to manage memory usage.

Findings

Reduces training time by up to 70% compared to traditional methods.

Produces better or comparable generalization performance across NLP tasks.

Effectively manages memory constraints with K-nearest neighbors strategy.

Abstract

Adversarial regularization can improve model generalization in many natural language processing tasks. However, conventional approaches are computationally expensive since they need to generate a perturbation for each sample in each epoch. We propose a new adversarial regularization method ARCH (adversarial regularization with caching), where perturbations are generated and cached once every several epochs. As caching all the perturbations imposes memory usage concerns, we adopt a K-nearest neighbors-based strategy to tackle this issue. The strategy only requires caching a small amount of perturbations, without introducing additional training time. We evaluate our proposed method on a set of neural machine translation and natural language understanding tasks. We observe that ARCH significantly eases the computational burden (saves up to 70% of computational time in comparison with conventional approaches). More surprisingly, by reducing the variance of stochastic gradients, ARCH produces a notably better (in most of the tasks) or comparable model generalization. Our code is available at https://github.com/SimiaoZuo/Caching-Adv.