EyeSim-VQA: A Free-Energy-Guided Eye Simulation Framework for Video Quality Assessment

TL;DR

EyeSimVQA introduces a free-energy-guided, dual-branch video quality assessment framework that enhances interpretability and performance by simulating adaptive repair behaviors and modeling gaze dynamics.

Contribution

It presents a novel VQA framework with free-energy-based self-repair, dual-branch architecture, and biologically inspired gaze modeling, addressing challenges in temporal dynamics and model stability.

Findings

Achieves competitive or superior performance on five VQA benchmarks.

Improves interpretability through biologically grounded design.

Effectively integrates enhancement modules without destabilizing the backbone.

Abstract

Free-energy-guided self-repair mechanisms have shown promising results in image quality assessment (IQA), but remain under-explored in video quality assessment (VQA), where temporal dynamics and model constraints pose unique challenges. Unlike static images, video content exhibits richer spatiotemporal complexity, making perceptual restoration more difficult. Moreover, VQA systems often rely on pre-trained backbones, which limits the direct integration of enhancement modules without affecting model stability. To address these issues, we propose EyeSimVQA, a novel VQA framework that incorporates free-energy-based self-repair. It adopts a dual-branch architecture, with an aesthetic branch for global perceptual evaluation and a technical branch for fine-grained structural and semantic analysis. Each branch integrates specialized enhancement modules tailored to distinct visual inputs-resized full-frame images and patch-based fragments-to simulate adaptive repair behaviors. We also explore a principled strategy for incorporating high-level visual features without disrupting the original backbone. In addition, we design a biologically inspired prediction head that models sweeping gaze dynamics to better fuse global and local representations for quality prediction. Experiments on five public VQA benchmarks demonstrate that EyeSimVQA achieves competitive or superior performance compared to state-of-the-art methods, while offering improved interpretability through its biologically grounded design.