Revealing Behavioral Plasticity in Large Language Models: A Token-Conditional Perspective

TL;DR

This paper uncovers that large language models can adapt their behavior dynamically through token-conditional prompts and stabilizes these adaptations using reinforcement learning, enabling flexible and controlled behavior changes at inference time.

Contribution

It introduces Token-Conditioned Reinforcement Learning (ToCoRL), a novel framework that internalizes behavioral plasticity in LLMs, allowing stable and targeted behavioral modifications without retraining.

Findings

ToCoRL enables precise behavioral control in LLMs.

Models can switch behaviors like reasoning and direct answering seamlessly.

Factual question answering improves without degrading capabilities.

Abstract

In this work, we reveal that Large Language Models (LLMs) possess intrinsic behavioral plasticity-akin to chameleons adapting their coloration to environmental cues-that can be exposed through token-conditional generation and stabilized via reinforcement learning. Specifically, by conditioning generation on carefully selected token prefixes sampled from responses exhibiting desired behaviors, LLMs seamlessly adapt their behavioral modes at inference time (e.g., switching from step-by-step reasoning to direct answering) without retraining. Based on this insight, we propose Token-Conditioned Reinforcement Learning (ToCoRL), a principled framework that leverages RL to internalize this chameleon-like plasticity, transforming transient inference-time adaptations into stable and learnable behavioral patterns. ToCoRL guides exploration with token-conditional generation and keep enhancing exploitation, enabling emergence of appropriate behaviors. Extensive experiments show that ToCoRL enables precise behavioral control without capability degradation. Notably, we show that large reasoning models, while performing strongly on complex mathematics, can be effectively adapted to excel at factual question answering, which was a capability previously hindered by their step-by-step reasoning patterns.