Do Models See in Line with Human Vision? Probing the Correspondence Between LVLM Representations and EEG Signals

TL;DR

This study investigates how well large vision language models' internal representations align with human EEG signals, revealing that certain layers and architectures show significant brain-like visual processing patterns, offering a new neuro-inspired evaluation benchmark.

Contribution

The paper introduces a novel method to quantify LVLM-brain alignment using EEG signals, highlighting the influence of architecture over scale and linking model performance to neural similarity.

Findings

Intermediate layers align with EEG in 100-300 ms window

Architectural design impacts brain alignment more than scale

Models with better visual performance show higher EEG similarity

Abstract

Large Vision Language Models (LVLMs) exhibit strong visual understanding and reasoning abilities. However, whether their internal representations reflect human visual cognition is still under-explored. In this paper, we address this by quantifying LVLM-brain alignment using image-evoked Electroencephalogram (EEG) signals, analyzing the effects of model architecture, scale, and image type. Specifically, by using ridge regression and representational similarity analysis, we compare visual representations from 32 open-source LVLMs with corresponding EEG responses. We observe a structured LVLM-brain correspondence: First, intermediate layers (8-16) show peak alignment with EEG activity in the 100-300 ms window, consistent with hierarchical human visual processing. Secondly, multimodal architectural design contributes 3.4 more to brain alignment than parameter scaling, and models with stronger downstream visual performance exhibit higher EEG similarity. Thirdly, spatiotemporal patterns further align with known cortical visual pathways. These results demonstrate that LVLMs learn human-aligned visual representations and establish neural alignment as a biologically grounded benchmark for evaluating and improving LVLMs. In addition, those results could provide insights that may inform the development of neuro-inspired applications.