ConPress: Learning Efficient Reasoning from Multi-Question Contextual Pressure

TL;DR

ConPress leverages a phenomenon called Self-Compression, where multi-question prompts lead models to produce shorter reasoning traces, enabling efficient fine-tuning that reduces inference costs without sacrificing accuracy.

Contribution

The paper introduces ConPress, a novel self-supervised fine-tuning method that internalizes self-compression behavior from multi-question prompts to improve reasoning efficiency.

Findings

Reduces reasoning token usage by 59% on MATH500

Achieves 33% reduction on AIME25

Maintains competitive accuracy with fewer tokens

Abstract

Large reasoning models (LRMs) typically solve reasoning-intensive tasks by generating long chain-of-thought (CoT) traces, leading to substantial inference overhead. We identify a reproducible inference-time phenomenon, termed Self-Compression: when multiple independent and answerable questions are presented within a single prompt, the model spontaneously produces shorter reasoning traces for each question. This phenomenon arises from multi-question contextual pressure during generation and consistently manifests across models and benchmarks. Building on this observation, we propose ConPress (Learning from Contextual Pressure), a lightweight self-supervised fine-tuning approach. ConPress constructs multi-question prompts to induce self-compression, samples the resulting model outputs, and parses and filters per-question traces to obtain concise yet correct reasoning trajectories. These trajectories are directly used for supervised fine-tuning, internalizing compressed reasoning behavior in single-question settings without external teachers, manual pruning, or reinforcement learning. With only 8k fine-tuning examples, ConPress reduces reasoning token usage by 59% on MATH500 and 33% on AIME25, while maintaining competitive accuracy.