VarMAE: Pre-training of Variational Masked Autoencoder for Domain-adaptive Language Understanding

TL;DR

VarMAE is a novel Transformer-based model that improves domain-specific language understanding by encoding context uncertainty into a smooth latent distribution, enabling effective adaptation with limited data.

Contribution

Introduces VarMAE, a masked autoencoder with a context uncertainty module for better domain adaptation in language models.

Findings

Effective adaptation to science and finance domains

Outperforms existing methods with limited domain data

Produces diverse, well-formed contextual representations

Abstract

Pre-trained language models have achieved promising performance on general benchmarks, but underperform when migrated to a specific domain. Recent works perform pre-training from scratch or continual pre-training on domain corpora. However, in many specific domains, the limited corpus can hardly support obtaining precise representations. To address this issue, we propose a novel Transformer-based language model named VarMAE for domain-adaptive language understanding. Under the masked autoencoding objective, we design a context uncertainty learning module to encode the token's context into a smooth latent distribution. The module can produce diverse and well-formed contextual representations. Experiments on science- and finance-domain NLU tasks demonstrate that VarMAE can be efficiently adapted to new domains with limited resources.