An Interactive Annotation Tool for Perceptual Video Compression

TL;DR

This paper introduces an interactive web-based tool for collecting fine-grained human feedback on perceptual video quality, enabling improved compression by visualizing and refining importance maps at a fixed bitrate.

Contribution

It presents a novel interactive annotation tool for spatio-temporal importance maps, facilitating perceptual quality optimization in video compression.

Findings

Users can iteratively refine importance maps to improve perceptual quality.

Videos annotated with the tool are 1.9 times more likely to be preferred at the same bitrate.

The tool effectively captures human perceptual preferences for video regions.

Abstract

Human perception is at the core of lossy video compression and yet, it is challenging to collect data that is sufficiently dense to drive compression. In perceptual quality assessment, human feedback is typically collected as a single scalar quality score indicating preference of one distorted video over another. In reality, some videos may be better in some parts but not in others. We propose an approach to collecting finer-grained feedback by asking users to use an interactive tool to directly optimize for perceptual quality given a fixed bitrate. To this end, we built a novel web-tool which allows users to paint these spatio-temporal importance maps over videos. The tool allows for interactive successive refinement: we iteratively re-encode the original video according to the painted importance maps, while maintaining the same bitrate, thus allowing the user to visually see the trade-off of assigning higher importance to one spatio-temporal part of the video at the cost of others. We use this tool to collect data in-the-wild (10 videos, 17 users) and utilize the obtained importance maps in the context of x264 coding to demonstrate that the tool can indeed be used to generate videos which, at the same bitrate, look perceptually better through a subjective study - and are 1.9 times more likely to be preferred by viewers. The code for the tool and dataset can be found at https://github.com/jenyap/video-annotation-tool.git