Prediction of Search Targets From Fixations in Open-World Settings

TL;DR

This paper introduces a novel approach for predicting visual search targets in open-world scenarios using fixation data, extending beyond traditional closed-world assumptions and demonstrating promising results with a new dataset.

Contribution

It presents the first open-world search target prediction framework based on fixation-target compatibilities and provides a new dataset for this task.

Findings

Closed-world baseline achieves accurate predictions with candidate sets of five images.

Proposed open-world formulation models fixation-target compatibilities.

Dataset includes fixation data from 18 users across three image categories.

Abstract

Previous work on predicting the target of visual search from human fixations only considered closed-world settings in which training labels are available and predictions are performed for a known set of potential targets. In this work we go beyond the state of the art by studying search target prediction in an open-world setting in which we no longer assume that we have fixation data to train for the search targets. We present a dataset containing fixation data of 18 users searching for natural images from three image categories within synthesised image collages of about 80 images. In a closed-world baseline experiment we show that we can predict the correct target image out of a candidate set of five images. We then present a new problem formulation for search target prediction in the open-world setting that is based on learning compatibilities between fixations and potential targets.