ReasonCACHE: Teaching LLMs To Reason Without Weight Updates

TL;DR

ReasonCACHE enables large language models to learn reasoning skills without weight updates by distilling demonstrations into a fixed key-value cache, surpassing traditional in-context learning and matching in-weight learning performance efficiently.

Contribution

The paper introduces ReasonCACHE, a novel method using Prefix Tuning to teach LLMs reasoning without overloading the context window or updating weights, outperforming standard ICL and matching IWL.

Findings

ReasonCACHE outperforms standard ICL on reasoning benchmarks.

It matches or surpasses in-weight learning approaches.

It is more efficient in data, inference cost, and parameters.

Abstract

Can Large language models (LLMs) learn to reason without any weight update and only through in-context learning (ICL)? ICL is strikingly sample-efficient, often learning from only a handful of demonstrations, but complex reasoning tasks typically demand many training examples to learn from. However, naively scaling ICL by adding more demonstrations breaks down at this scale: attention costs grow quadratically, performance saturates or degrades with longer contexts, and the approach remains a shallow form of learning. Due to these limitations, practitioners predominantly rely on in-weight learning (IWL) to induce reasoning. In this work, we show that by using Prefix Tuning, LLMs can learn to reason without overloading the context window and without any weight updates. We introduce $\textbf{ReasonCACHE}$, an instantiation of this mechanism that distills demonstrations into a fixed key-value cache. Empirically, across challenging reasoning benchmarks, including GPQA-Diamond, ReasonCACHE outperforms standard ICL and matches or surpasses IWL approaches. Further, it achieves this all while being more efficient across three key axes: data, inference cost, and trainable parameters. We also theoretically prove that ReasonCACHE can be strictly more expressive than low-rank weight update since the latter ties expressivity to input rank, whereas ReasonCACHE bypasses this constraint by directly injecting key-values into the attention mechanism. Together, our findings identify ReasonCACHE as a middle path between in-context and in-weight learning, providing a scalable algorithm for learning reasoning skills beyond the context window without modifying parameters. Our project page: https://reasoncache.github.io/