Incentivizing Dual Process Thinking for Efficient Large Language Model Reasoning

TL;DR

This paper introduces ACPO, a reinforcement learning framework inspired by dual process theory, enabling large reasoning models to adaptively switch thinking modes for more efficient and less redundant reasoning.

Contribution

It proposes a novel adaptive reasoning method with explicit thinking modes and difficulty-aware switching, improving reasoning efficiency in large models.

Findings

Reduces redundant reasoning in large models

Enhances adaptive cognitive switching based on task difficulty

Achieves more efficient hybrid reasoning

Abstract

Large reasoning models (LRMs) have demonstrated strong performance on complex reasoning tasks, but often suffer from overthinking, generating redundant content regardless of task difficulty. Inspired by the dual process theory in cognitive science, we propose Adaptive Cognition Policy Optimization (ACPO), a reinforcement learning framework that enables LRMs to achieve efficient reasoning through adaptive cognitive allocation and dynamic system switch. ACPO incorporates two key components: (1) introducing system-aware reasoning tokens to explicitly represent the thinking modes thereby making the model's cognitive process transparent, and (2) integrating online difficulty estimation and token length budget to guide adaptive system switch and reasoning during reinforcement learning. To this end, we propose a two-stage training strategy. The first stage begins with supervised fine-tuning to cold start the model, enabling it to generate reasoning paths with explicit thinking modes. In the second stage, we apply ACPO to further enhance adaptive system switch for difficulty-aware reasoning. Experimental results demonstrate that ACPO effectively reduces redundant reasoning while adaptively adjusting cognitive allocation based on task complexity, achieving efficient hybrid reasoning.