EVA: Bridging Performance and Human Alignment in Hard-Attention Vision Models for Image Classification

TL;DR

EVA is a neuroscience-inspired hard-attention model that balances classification accuracy with human-like scanpaths, improving interpretability in vision tasks without extensive gaze supervision.

Contribution

Introduces EVA, a novel attention mechanism that explicitly manages the trade-off between performance and human alignment in vision models, inspired by neuroscience.

Findings

EVA improves scanpath alignment on CIFAR-10 with minimal accuracy loss.

Variance control and gating restore human-like trajectories suppressed by CNN features.

EVA achieves human-like scanpaths on ImageNet-100 and COCO-Search18 without additional gaze supervision.

Abstract

Optimizing vision models purely for classification accuracy can impose an alignment tax, degrading human-like scanpaths and limiting interpretability. We introduce EVA, a neuroscience-inspired hard-attention mechanistic testbed that makes the performance-human-likeness trade-off explicit and adjustable. EVA samples a small number of sequential glimpses using a minimal fovea-periphery representation with CNN-based feature extractor and integrates variance control and adaptive gating to stabilize and regulate attention dynamics. EVA is trained with the standard classification objective without gaze supervision. On CIFAR-10 with dense human gaze annotations, EVA improves scanpath alignment under established metrics such as DTW, NSS, while maintaining competitive accuracy. Ablations show that CNN-based feature extraction drives accuracy but suppresses human-likeness, whereas variance control and gating restore human-aligned trajectories with minimal performance loss. We further validate EVA's scalability on ImageNet-100 and evaluate scanpath alignment on COCO-Search18 without COCO-Search18 gaze supervision or finetuning, where EVA yields human-like scanpaths on natural scenes without additional training. Overall, EVA provides a principled framework for trustworthy, human-interpretable active vision.