A deep Q-learning method for optimizing visual search strategies in backgrounds of dynamic noise

TL;DR

This paper explores using deep Q-learning to approximate the ideal searcher strategy for visual search tasks in complex backgrounds, aiming to improve understanding and performance of human visual search behavior.

Contribution

It demonstrates that a Q-network-based reinforcement learning method can effectively approximate the Bayesian ideal searcher in complex, realistic backgrounds, extending prior work on simpler noise models.

Findings

Q-network search strategy aligns with the Bayesian ideal searcher

Reinforcement learning can handle complex, realistic backgrounds

Potential to improve human visual search strategies

Abstract

Humans process visual information with varying resolution (foveated visual system) and explore images by orienting through eye movements the high-resolution fovea to points of interest. The Bayesian ideal searcher (IS) that employs complete knowledge of task-relevant information optimizes eye movement strategy and achieves the optimal search performance. The IS can be employed as an important tool to evaluate the optimality of human eye movements, and potentially provide guidance to improve human observer visual search strategies. Najemnik and Geisler (2005) derived an IS for backgrounds of spatial 1/f noise. The corresponding template responses follow Gaussian distributions and the optimal search strategy can be analytically determined. However, the computation of the IS can be intractable when considering more realistic and complex backgrounds such as medical images. Modern reinforcement learning methods, successfully applied to obtain optimal policy for a variety of tasks, do not require complete knowledge of the background generating functions and can be potentially applied to anatomical backgrounds. An important first step is to validate the optimality of the reinforcement learning method. In this study, we investigate the ability of a reinforcement learning method that employs Q-network to approximate the IS. We demonstrate that the search strategy corresponding to the Q-network is consistent with the IS search strategy. The findings show the potential of the reinforcement learning with Q-network approach to estimate optimal eye movement planning with real anatomical backgrounds.