TimeHC-RL: Temporal-aware Hierarchical Cognitive Reinforcement Learning for Enhancing LLMs' Social Intelligence

TL;DR

This paper introduces TimeHC-RL, a novel hierarchical reinforcement learning approach that enhances LLMs' social intelligence by incorporating temporal and cognitive mode considerations, outperforming existing methods.

Contribution

The paper proposes a new temporal-aware hierarchical RL framework specifically designed to improve LLMs' social cognition, addressing a gap in post-training social intelligence enhancement.

Findings

TimeHC-RL outperforms traditional System 2 RL methods.

A 7B model with TimeHC-RL rivals larger models like DeepSeek-R1 and OpenAI-O3.

Systematic exploration reveals key insights into post-training and test-time interventions.

Abstract

Recently, Large Language Models (LLMs) have made significant progress in IQ-related domains that require careful thinking, such as mathematics and coding. However, enhancing LLMs' cognitive development in social domains, particularly from a post-training perspective, remains underexplored. Recognizing that the social world follows a distinct timeline and requires a richer blend of cognitive modes (from intuitive reactions (System 1) and surface-level thinking to deliberate thinking (System 2)) than mathematics, which primarily relies on System 2 cognition (careful, step-by-step reasoning), we introduce Temporal-aware Hierarchical Cognitive Reinforcement Learning (TimeHC-RL) for enhancing LLMs' social intelligence. In our experiments, we systematically explore improving LLMs' social intelligence and validate the effectiveness of the TimeHC-RL method, through five other post-training paradigms and two test-time intervention paradigms on eight datasets with diverse data patterns. Experimental results reveal the superiority of our proposed TimeHC-RL method compared to the widely adopted System 2 RL method. It gives the 7B backbone model wings, enabling it to rival the performance of advanced models like DeepSeek-R1 and OpenAI-O3. Additionally, the systematic exploration from post-training and test-time interventions perspectives to improve LLMs' social intelligence has uncovered several valuable insights.