An Analysis and Mitigation of the Reversal Curse
Ang Lv, Kaiyi Zhang, Shufang Xie, Quan Tu, Yuhan Chen and, Ji-Rong Wen, Rui Yan

TL;DR
This paper investigates the reversal curse in large language models, a phenomenon where models perform well on forward relations but struggle with their inverses, revealing limitations linked to training objectives.
Contribution
First comprehensive analysis of the reversal curse in LLMs, linking it to training objectives and highlighting a key limitation in current models.
Findings
Reversal curse stems from next-token prediction training
Models excel in forward relation tasks but struggle with inverse relations
Highlights need for improved training strategies to address this limitation
Abstract
Recent research observed a noteworthy phenomenon in large language models (LLMs), referred to as the ``reversal curse.'' The reversal curse is that when dealing with two entities, denoted as and , connected by their relation and its inverse , LLMs excel in handling sequences in the form of ``,'' but encounter challenges when processing ``,'' whether in generation or comprehension. For instance, GPT-4 can accurately respond to the query ``Tom Cruise's mother is?'' with ``Mary Lee Pfeiffer,'' but it struggles to provide a satisfactory answer when asked ``Mary Lee Pfeiffer's son is?'' In this paper, we undertake the first-ever study of how the reversal curse happens in LLMs. Our investigations reveal that the reversal curse can stem from the specific training objectives, which become particularly evident in the widespread use of next-token prediction…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
Taxonomy
TopicsTopic Modeling · Natural Language Processing Techniques · Text Readability and Simplification
MethodsFocus · GLM
