Enhancing Latent Computation in Transformers with Latent Tokens

TL;DR

This paper introduces latent tokens, a lightweight augmentation for Transformer-based language models that improves performance and out-of-distribution generalization by steering decoding through attention mechanisms.

Contribution

The paper proposes a novel, parameter-efficient method called latent tokens to enhance LLMs, seamlessly integrating with pre-trained models and improving adaptability.

Findings

Latent tokens significantly outperform baselines in out-of-distribution tasks.

The method can be integrated with pre-trained Transformers with minimal overhead.

Synthetic tasks verify the hypotheses about latent tokens' mechanisms.

Abstract

Augmenting large language models (LLMs) with auxiliary tokens has emerged as a promising strategy for enhancing model performance. In this work, we introduce a lightweight method termed latent tokens; these are dummy tokens that may be non-interpretable in natural language but steer the autoregressive decoding process of a Transformer-based LLM via the attention mechanism. The proposed latent tokens can be seamlessly integrated with a pre-trained Transformer, trained in a parameter-efficient manner, and applied flexibly at inference time, while adding minimal complexity overhead to the existing infrastructure of standard Transformers. We propose several hypotheses about the underlying mechanisms of latent tokens and design synthetic tasks accordingly to verify them. Numerical results confirm that the proposed method noticeably outperforms the baselines, particularly in the out-of-distribution generalization scenarios, highlighting its potential in improving the adaptability of LLMs.