Causal Reasoning Favors Encoders: On The Limits of Decoder-Only Models

TL;DR

This paper investigates the effectiveness of encoder-based models versus decoder-only models for causal reasoning, finding that encoder architectures with finetuning outperform decoder-only models in robustness and generalization for multihop reasoning tasks.

Contribution

It demonstrates that encoder and encoder-decoder architectures are better suited for causal reasoning than decoder-only models, especially when finetuned, highlighting their advantages in robustness and generalization.

Findings

Encoder models outperform decoder-only models in causal reasoning tasks.

Finetuned encoder architectures generalize better across distributional shifts.

Decoder-only models excel only at large scales, but are less robust.

Abstract

In context learning (ICL) underpins recent advances in large language models (LLMs), although its role and performance in causal reasoning remains unclear. Causal reasoning demands multihop composition and strict conjunctive control, and reliance on spurious lexical relations of the input could provide misleading results. We hypothesize that, due to their ability to project the input into a latent space, encoder and encoder decoder architectures are better suited for said multihop conjunctive reasoning versus decoder only models. To do this, we compare fine-tuned versions of all the aforementioned architectures with zero and few shot ICL in both natural language and non natural language scenarios. We find that ICL alone is insufficient for reliable causal reasoning, often overfocusing on irrelevant input features. In particular, decoder only models are noticeably brittle to distributional shifts, while finetuned encoder and encoder decoder models can generalize more robustly across our tests, including the non natural language split. Both architectures are only matched or surpassed by decoder only architectures at large scales. We conclude by noting that for cost effective, short horizon robust causal reasoning, encoder or encoder decoder architectures with targeted finetuning are preferable.