Human Vision Constrained Super-Resolution

TL;DR

This paper introduces a human vision-inspired framework that adaptively guides super-resolution methods based on human visual sensitivity, improving computational efficiency without compromising perceived image quality.

Contribution

It proposes an architecture-agnostic Human Visual Processing Framework (HVPF) that dynamically adjusts super-resolution processing according to human visual perception and viewing conditions.

Findings

Reduces FLOPS by 2x or more while maintaining quality.

Demonstrates effectiveness through quantitative and qualitative evaluations.

User studies confirm perceived quality is preserved.

Abstract

Modern deep-learning super-resolution (SR) techniques process images and videos independently of the underlying content and viewing conditions. However, the sensitivity of the human visual system (HVS) to image details changes depending on the underlying image characteristics, such as spatial frequency, luminance, color, contrast, or motion; as well viewing condition aspects such as ambient lighting and distance to the display. This observation suggests that computational resources spent on up-sampling images/videos may be wasted whenever a viewer cannot resolve the synthesized details i.e the resolution of details exceeds the resolving capability of human vision. Motivated by this observation, we propose a human vision inspired and architecture-agnostic approach for controlling SR techniques to deliver visually optimal results while limiting computational complexity. Its core is an explicit Human Visual Processing Framework (HVPF) that dynamically and locally guides SR methods according to human sensitivity to specific image details and viewing conditions. We demonstrate the application of our framework in combination with network branching to improve the computational efficiency of SR methods. Quantitative and qualitative evaluations, including user studies, demonstrate the effectiveness of our approach in reducing FLOPS by factors of 2$\times$ and greater, without sacrificing perceived quality.