Contextual Representation Learning beyond Masked Language Modeling

TL;DR

This paper analyzes how masked language models learn representations and introduces TACO, a new method that models global semantics to improve efficiency and performance, achieving significant speedups and accuracy gains.

Contribution

The paper proposes TACO, a novel approach that directly models global semantics in contextual representations, addressing limitations of traditional MLMs.

Findings

TACO achieves up to 5x speedup over existing MLMs.

TACO improves average GLUE scores by up to 1.2 points.

TACO effectively aligns contextual semantics for better representations.

Abstract

How do masked language models (MLMs) such as BERT learn contextual representations? In this work, we analyze the learning dynamics of MLMs. We find that MLMs adopt sampled embeddings as anchors to estimate and inject contextual semantics to representations, which limits the efficiency and effectiveness of MLMs. To address these issues, we propose TACO, a simple yet effective representation learning approach to directly model global semantics. TACO extracts and aligns contextual semantics hidden in contextualized representations to encourage models to attend global semantics when generating contextualized representations. Experiments on the GLUE benchmark show that TACO achieves up to 5x speedup and up to 1.2 points average improvement over existing MLMs. The code is available at https://github.com/FUZHIYI/TACO.