FOVI: A biologically-inspired foveated interface for deep vision models

TL;DR

This paper introduces FOVI, a biologically-inspired foveated interface for deep vision models that improves efficiency by mimicking human eye movement and resolution variation, enabling scalable high-resolution vision processing.

Contribution

The paper presents a novel foveated vision interface (FOVI) that reformats variable-resolution sensor data into a dense, V1-like manifold, enabling efficient convolutional architectures and adaptation of existing models.

Findings

FOVI achieves competitive performance with reduced computational cost.

The approach enables scalable high-resolution egocentric vision.

Code and models are publicly available.

Abstract

Human vision is foveated, with variable resolution peaking at the center of a large field of view; this reflects an efficient trade-off for active sensing, allowing eye-movements to bring different parts of the world into focus with other parts of the world in context. In contrast, most computer vision systems encode the visual world at a uniform resolution, raising challenges for processing full-field high-resolution images efficiently. We propose a foveated vision interface (FOVI) based on the human retina and primary visual cortex, that reformats a variable-resolution retina-like sensor array into a uniformly dense, V1-like sensor manifold. Receptive fields are defined as k-nearest-neighborhoods (kNNs) on the sensor manifold, enabling kNN-convolution via a novel kernel mapping technique. We demonstrate two use cases: (1) an end-to-end kNN-convolutional architecture, and (2) a foveated adaptation of the foundational DINOv3 ViT model, leveraging low-rank adaptation (LoRA). These models provide competitive performance at a fraction of the computational cost of non-foveated baselines, opening pathways for efficient and scalable active sensing for high-resolution egocentric vision. Code and pre-trained models are available at https://github.com/nblauch/fovi and https://huggingface.co/fovi-pytorch.