Affinity-based Attention in Self-supervised Transformers Predicts Dynamics of Object Grouping in Humans

TL;DR

This paper introduces a model of human-like object grouping based on attention spreading in self-supervised vision Transformers, demonstrating improved prediction of human reaction times in natural images without task-specific training.

Contribution

It proposes a novel affinity-based attention spreading mechanism in Transformers for modeling human object segmentation and provides new benchmarks for visual representation learning.

Findings

Models outperform CNN baselines in predicting human reaction times

Self-supervised Transformer features effectively model attention spreading

No task-specific training was needed for the models to predict human behavior

Abstract

The spreading of attention has been proposed as a mechanism for how humans group features to segment objects. However, such a mechanism has not yet been implemented and tested in naturalistic images. Here, we leverage the feature maps from self-supervised vision Transformers and propose a model of human object-based attention spreading and segmentation. Attention spreads within an object through the feature affinity signal between different patches of the image. We also collected behavioral data on people grouping objects in natural images by judging whether two dots are on the same object or on two different objects. We found that our models of affinity spread that were built on feature maps from the self-supervised Transformers showed significant improvement over baseline and CNN based models on predicting reaction time patterns of humans, despite not being trained on the task or with any other object labels. Our work provides new benchmarks for evaluating models of visual representation learning including Transformers.