Gaze patterns predict preference and confidence in pairwise AI image evaluation

TL;DR

This study shows that eye-tracking can predict preferences and confidence levels during pairwise evaluation of AI-generated images, revealing underlying cognitive processes.

Contribution

It demonstrates that gaze patterns can predict choices and confidence in pairwise image evaluation, providing insights into preference formation.

Findings

Gaze shifts toward chosen images before decision by about one second.

Gaze features predict binary choice with 68% accuracy.

Gaze transitions differentiate high-confidence from uncertain decisions with 66% accuracy.

Abstract

Preference learning methods, such as Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO), rely on pairwise human judgments, yet little is known about the cognitive processes underlying these judgments. We investigate whether eye-tracking can reveal preference formation during pairwise AI-generated image evaluation. Thirty participants completed 1,800 trials while their gaze was recorded. We replicated the gaze cascade effect, with gaze shifting toward chosen images approximately one second before the decision. Cascade dynamics were consistent across confidence levels. Gaze features predicted binary choice (68% accuracy), with chosen images receiving more dwell time, fixations, and revisits. Gaze transitions distinguished high-confidence from uncertain decisions (66% accuracy), with low-confidence trials showing more image switches per second. These results show that gaze patterns predict both choice and confidence in pairwise image evaluations, suggesting that eye-tracking provides implicit signals relevant to the quality of preference annotations.