Breaking the Reversal Curse in Autoregressive Language Models via Identity Bridge

TL;DR

This paper demonstrates that the reversal curse in autoregressive language models can be mitigated by a simple data regularization technique called the Identity Bridge, enabling models to learn higher-level rules rather than just memorizing facts.

Contribution

The paper introduces the Identity Bridge data recipe and provides theoretical and empirical evidence that it helps models overcome the reversal curse, a fundamental limitation in LLMs.

Findings

A 1B model finetuned with the recipe achieves 40% success on reversal tasks.

Without the recipe, models have near-zero success in reversal tasks.

Theoretical analysis shows even one-layer transformers can break the reversal curse with this method.

Abstract

Autoregressive large language models (LLMs) have achieved remarkable success in many complex tasks, yet they can still fail in very simple logical reasoning such as the "reversal curse" -- when trained on forward knowledge data of the form "$A \rightarrow B$" (e.g., Alice's husband is Bob), the model is unable to deduce the reversal knowledge "$B \leftarrow A$" (e.g., Bob's wife is Alice) during test. Extensive prior research suggests that this failure is an inherent, fundamental limit of autoregressive causal LLMs, indicating that these models tend to memorize factual-level knowledge rather than capture higher-level rules. In this paper, we challenge this view by showing that this seemingly fundamental limit can be mitigated by slightly tweaking the training data with a simple regularization data recipe called the Identity Bridge of the form "$A \to A$" (e.g., The name of Alice is Alice). Theoretically, we prove that under this recipe, even a one-layer transformer can break the reversal curse by analyzing the implicit bias of gradient descent. Empirically, we show that a 1B pretrained language model finetuned with the proposed data recipe achieves a 40% success rate on reversal tasks, in stark contrast to a near-zero success rate when trained solely on forward-knowledge data. Our work provides a novel theoretical foundation for the reversal curse and offers a principled, low-cost path to encouraging LLMs to learn higher-level rules from data.