Cognitive Chain-of-Thought (CoCoT): Structured Multimodal Reasoning about Social Situations

TL;DR

The paper introduces CoCoT, a structured reasoning framework for multimodal social tasks, improving model interpretability and performance by mimicking cognitive stages of perception, situation understanding, and norm application.

Contribution

It proposes a novel three-stage reasoning framework for multimodal social tasks, enhancing model performance and interpretability without explicit prompting at inference.

Findings

CoCoT improves task accuracy by 4.6% to 5.9% across multiple social reasoning tasks.

Supervised fine-tuning on CoCoT traces yields 5-6% performance gains.

Structured reasoning enhances model interpretability and social alignment.

Abstract

Chain-of-Thought (CoT) prompting helps models think step by step. But naive CoT breaks down in visually grounded social tasks, where models must perceive, understand, and judge all at once; bridging perception with norm-grounded reasoning. Recent work has introduced structured reasoning for multi-turn agent planning and visual QA, decomposing tasks into sequential sub-goals. To extend this to single-shot multimodal social reasoning, we introduce Cognitive Chain-of-Thought (CoCoT), a reasoning framework that structures vision-language-model (VLM) reasoning through three cognitively inspired stages: Perception (extract grounded facts), Situation (infer situations), and Norm (applying social norms). Evaluation across multiple distinct tasks such as multimodal intent disambiguation, multimodal theory of mind, social commonsense reasoning, and safety instruction following, shows consistent improvements (5.9% to 4.6% on average). We further explore the utility of CoCoT for improving models' reasoning through training and show that supervised fine-tuning on CoCoT-structured traces yields 5-6% improvements without explicit CoCoT prompting at inference, demonstrating that models internalize the structured reasoning pattern rather than merely following instructions. We show that structuring model reasoning through cognitively grounded stages enhances interpretability and social alignment, laying the groundwork for more reliable multimodal systems. All code and data will be released publicly.