Benchmarking and Understanding Compositional Relational Reasoning of LLMs

TL;DR

This paper introduces a new benchmark called GAR to evaluate the compositional relational reasoning capabilities of large language models, revealing their fundamental deficiencies and identifying core attention heads involved in reasoning.

Contribution

It proposes a synthetic benchmark for CRR, systematically analyzes LLMs' reasoning circuits, and identifies key attention heads crucial for CRR tasks.

Findings

Existing LLMs struggle with CRR tasks in GAR benchmark.

Core attention heads are vital for reasoning in LLMs.

Two classes of heads encode true and false concepts in GAR.

Abstract

Compositional relational reasoning (CRR) is a hallmark of human intelligence, but we lack a clear understanding of whether and how existing transformer large language models (LLMs) can solve CRR tasks. To enable systematic exploration of the CRR capability of LLMs, we first propose a new synthetic benchmark called Generalized Associative Recall (GAR) by integrating and generalizing the essence of several tasks in mechanistic interpretability (MI) study in a unified framework. Evaluation shows that GAR is challenging enough for existing LLMs, revealing their fundamental deficiency in CRR. Meanwhile, it is easy enough for systematic MI study. Then, to understand how LLMs solve GAR tasks, we use attribution patching to discover the core circuits reused by Vicuna-33B across different tasks and a set of vital attention heads. Intervention experiments show that the correct functioning of these heads significantly impacts task performance. Especially, we identify two classes of heads whose activations represent the abstract notion of true and false in GAR tasks respectively. They play a fundamental role in CRR across various models and tasks. The dataset and code are available at https://github.com/Caiyun-AI/GAR.