CR-Eyes: A Computational Rational Model of Visual Sampling Behavior in Atari Games

TL;DR

CR-Eyes is a reinforcement learning-based computational model that simulates human-like visual sampling and decision-making in Atari games, accounting for perceptual and cognitive constraints.

Contribution

It introduces a novel, goal-directed visual sampling model trained via reinforcement learning that explicitly integrates perception and action in pixel-based environments.

Findings

CR-Eyes aligns well with human task performance and saliency patterns.

The model reveals systematic differences in scanpaths compared to humans.

It demonstrates the feasibility of scalable, theory-grounded user models for interactive system design.

Abstract

Designing mobile and interactive technologies requires understanding how users sample dynamic environments to acquire information and make decisions under time pressure. However, existing computational user models either rely on hand-crafted task representations or are limited to static or non-interactive visual inputs, restricting their applicability to realistic, pixel-based environments. We present CR-Eyes, a computationally rational model that simulates visual sampling and gameplay behavior in Atari games. Trained via reinforcement learning, CR-Eyes operates under perceptual and cognitive constraints and jointly learns where to look and how to act in a time-sensitive setting. By explicitly closing the perception-action loop, the model treats eye movements as goal-directed actions rather than as isolated saliency predictions. Our evaluation shows strong alignment with human data in task performance and aggregate saliency patterns, while also revealing systematic differences in scanpaths. CR-Eyes is a step toward scalable, theory-grounded user models that support design and evaluation of interactive systems.