Deep Models, Shallow Alignment: Uncovering the Granularity Mismatch in Neural Decoding

TL;DR

This paper introduces Shallow Alignment, a contrastive learning method that aligns neural signals with intermediate visual representations, improving neural decoding accuracy by addressing the granularity mismatch between neural signals and deep vision models.

Contribution

The paper proposes a novel contrastive learning strategy that aligns neural signals with intermediate visual representations, significantly enhancing decoding performance over standard methods.

Findings

Performance gains of 22% to 58% across benchmarks

Effective unlocking of the scaling law in neural decoding

Systematic analysis of underlying mechanisms

Abstract

Neural visual decoding is a central problem in brain computer interface research, aiming to reconstruct human visual perception and to elucidate the structure of neural representations. However, existing approaches overlook a fundamental granularity mismatch between human and machine vision, where deep vision models emphasize semantic invariance by suppressing local texture information, whereas neural signals preserve an intricate mixture of low-level visual attributes and high-level semantic content. To address this mismatch, we propose Shallow Alignment, a novel contrastive learning strategy that aligns neural signals with intermediate representations of visual encoders rather than their final outputs, thereby striking a better balance between low-level texture details and high-level semantic features. Extensive experiments across multiple benchmarks demonstrate that Shallow Alignment significantly outperforms standard final-layer alignment, with performance gains ranging from 22% to 58% across diverse vision backbones. Notably, our approach effectively unlocks the scaling law in neural visual decoding, enabling decoding performance to scale predictably with the capacity of pre-trained vision backbones. We further conduct systematic empirical analyses to shed light on the mechanisms underlying the observed performance gains.