Generic decoding of seen and imagined objects using hierarchical visual features

TL;DR

This study demonstrates a method to decode both seen and imagined objects from fMRI data using hierarchical visual features, revealing parallels between human and machine vision and enabling brain-based object identification.

Contribution

The paper introduces a novel decoding approach that predicts hierarchical visual features from fMRI data to identify objects, including imagined ones, beyond training examples.

Findings

Hierarchical features can be predicted from fMRI patterns.

Higher-level features are associated with higher visual areas.

Decoding reveals progressive recruitment of visual representations during imagination.

Abstract

Object recognition is a key function in both human and machine vision. While recent studies have achieved fMRI decoding of seen and imagined contents, the prediction is limited to training examples. We present a decoding approach for arbitrary objects, using the machine vision principle that an object category is represented by a set of features rendered invariant through hierarchical processing. We show that visual features including those from a convolutional neural network can be predicted from fMRI patterns and that greater accuracy is achieved for low/high-level features with lower/higher-level visual areas, respectively. Predicted features are used to identify seen/imagined object categories (extending beyond decoder training) from a set of computed features for numerous object images. Furthermore, the decoding of imagined objects reveals progressive recruitment of higher to lower visual representations. Our results demonstrate a homology between human and machine vision and its utility for brain-based information retrieval.