Visual Bias in Simulated Users: The Impact of Luminance and Contrast on Reinforcement Learning-based Interaction

TL;DR

This study systematically examines how luminance and contrast influence reinforcement learning-based simulated users in HCI tasks, revealing that visual artifacts significantly affect behavior and robustness, especially under static distractors.

Contribution

It is the first to analyze the impact of luminance and contrast on RL-trained simulated users, highlighting the importance of visual rendering choices in simulation validity.

Findings

Luminance critically affects performance with static distractors.

Motion cues improve robustness against luminance variations.

Extreme luminances like black can yield high performance but poor robustness.

Abstract

Reinforcement learning (RL) enables simulations of HCI tasks, yet their validity is questionable when performance is driven by visual rendering artifacts distinct from interaction design. We provide the first systematic analysis of how luminance and contrast affect behavior by training 247 \RV{simulated users using RL} on pointing and tracking tasks. We vary the luminance of task-relevant objects, distractors, and background under no distractor, static distractor, and moving distractor conditions, and evaluate task performance and robustness to unseen luminances. Results show luminance becomes critical with static distractors, substantially degrading performance and robustness, whereas motion cues mitigate this issue. Furthermore, robustness depends on preserving relational ordering between luminances rather than matching absolute values. Extreme luminances, especially black, often yield high performance but poor robustness. Overall, seemingly minor luminance changes can strongly shape learned behavior, revealing critical insights into what RL-driven simulated users actually learn.