Fusing Foveal Fixations Using Linear Retinal Transformations and Bayesian Experimental Design

TL;DR

This paper introduces a model that fuses multiple scene fixations into a unified representation by explicitly modeling retinal transformations and optimizing gaze decisions through Bayesian experimental design, demonstrated on image datasets.

Contribution

It presents a novel approach combining linear retinal transformations with Bayesian experimental design for scene fusion, enabling exact inference and optimized gaze planning.

Findings

Effective scene fusion demonstrated on Frey faces and MNIST datasets.

Exact inference achieved through linear retinal transformation modeling.

Bayesian experimental design improves gaze selection for scene understanding.

Abstract

Humans (and many vertebrates) face the problem of fusing together multiple fixations of a scene in order to obtain a representation of the whole, where each fixation uses a high-resolution fovea and decreasing resolution in the periphery. In this paper we explicitly represent the retinal transformation of a fixation as a linear downsampling of a high-resolution latent image of the scene, exploiting the known geometry. This linear transformation allows us to carry out exact inference for the latent variables in factor analysis (FA) and mixtures of FA models of the scene. Further, this allows us to formulate and solve the choice of "where to look next" as a Bayesian experimental design problem using the Expected Information Gain criterion. Experiments on the Frey faces and MNIST datasets demonstrate the effectiveness of our models.