Multigranular Evaluation for Brain Visual Decoding

TL;DR

This paper introduces BASIC, a multigranular evaluation framework for brain visual decoding that assesses structural, semantic, and contextual fidelity, enabling more detailed and interpretable comparisons of decoding methods.

Contribution

The paper presents a novel, unified evaluation framework that captures fine-grained visual distinctions and aligns with neuroscientific principles, improving upon coarse existing metrics.

Findings

BASIC provides more discriminative evaluation metrics.

Benchmarking reveals differences among decoding methods.

Framework enhances interpretability of decoding performance.

Abstract

Existing evaluation protocols for brain visual decoding predominantly rely on coarse metrics that obscure inter-model differences, lack neuroscientific foundation, and fail to capture fine-grained visual distinctions. To address these limitations, we introduce BASIC, a unified, multigranular evaluation framework that jointly quantifies structural fidelity, inferential alignment, and contextual coherence between decoded and ground-truth images. For the structural level, we introduce a hierarchical suite of segmentation-based metrics, including foreground, semantic, instance, and component masks, anchored in granularity-aware correspondence across mask structures. For the semantic level, we extract structured scene representations encompassing objects, attributes, and relationships using multimodal large language models, enabling detailed, scalable, and context-rich comparisons with ground-truth stimuli. We benchmark a diverse set of visual decoding methods across multiple stimulus-neuroimaging datasets within this unified evaluation framework. Together, these criteria provide a more discriminative, interpretable, and comprehensive foundation for evaluating brain visual decoding methods.