Toward Cognitive Supersensing in Multimodal Large Language Model

TL;DR

This paper introduces Cognitive Supersensing, a training paradigm for multimodal large language models that incorporates visual imagery capabilities to enhance complex cognitive reasoning, demonstrated by superior performance on a new VQA benchmark.

Contribution

It proposes a novel training method integrating visual latent prediction and reinforcement learning to improve cognitive reasoning in MLLMs, along with a comprehensive benchmark for evaluation.

Findings

Significant performance improvements on CogSense-Bench.

Enhanced generalization on out-of-domain VQA tasks.

Demonstrated importance of visual imagery in cognitive reasoning.

Abstract

Multimodal Large Language Models (MLLMs) have achieved remarkable success in open-vocabulary perceptual tasks, yet their ability to solve complex cognitive problems remains limited, especially when visual details are abstract and require visual memory. Current approaches primarily scale Chain-of-Thought (CoT) reasoning in the text space, even when language alone is insufficient for clear and structured reasoning, and largely neglect visual reasoning mechanisms analogous to the human visuospatial sketchpad and visual imagery. To mitigate this deficiency, we introduce Cognitive Supersensing, a novel training paradigm that endows MLLMs with human-like visual imagery capabilities by integrating a Latent Visual Imagery Prediction (LVIP) head that jointly learns sequences of visual cognitive latent embeddings and aligns them with the answer, thereby forming vision-based internal reasoning chains. We further introduce a reinforcement learning stage that optimizes text reasoning paths based on this grounded visual latent. To evaluate the cognitive capabilities of MLLMs, we present CogSense-Bench, a comprehensive visual question answering (VQA) benchmark assessing five cognitive dimensions. Extensive experiments demonstrate that MLLMs trained with Cognitive Supersensing significantly outperform state-of-the-art baselines on CogSense-Bench and exhibit superior generalization on out-of-domain mathematics and science VQA benchmarks, suggesting that internal visual imagery is potentially key to bridging the gap between perceptual recognition and cognitive understanding. We will open-source the CogSense-Bench and our model weights.