LACONIC: Length-Aware Constrained Reinforcement Learning for LLM

TL;DR

LACONIC is a reinforcement learning approach that effectively enforces length constraints on large language models, reducing output length by over 50% while maintaining or improving task performance across various benchmarks.

Contribution

It introduces a length-aware RL method with adaptive cost scaling that ensures robust length control without sacrificing task accuracy.

Findings

Reduces output length by over 50% on mathematical reasoning tasks.

Maintains out-of-domain performance with 44% fewer tokens.

Integrates seamlessly into existing RL-tuning pipelines.

Abstract

Reinforcement learning (RL) has enhanced the capabilities of large language models (LLMs) through reward-driven training. Nevertheless, this process can introduce excessively long responses, inflating inference latency and computational overhead. Prior length-control approaches typically rely on fixed heuristic reward shaping, which can misalign with the task objective and require brittle tuning. In this work, we propose LACONIC, a reinforcement learning method that enforces a target token budget during training. Specifically, we update policy models using an augmented objective that combines the task reward with a length-based cost. To balance brevity and task performance, the cost scale is adaptively adjusted throughout training. This yields robust length control while preserving task reward. We provide a theoretical guarantee that support the method. Across mathematical reasoning models and datasets, LACONIC preserves or improves pass@1 while reducing output length by over 50%. It maintains out-of-domain performance on general knowledge and multilingual benchmarks with 44% fewer tokens. Moreover, LACONIC integrates into standard RL-tuning with no inference changes and minimal deployment overhead.