Reinforced Bit Allocation under Task-Driven Semantic Distortion Metrics

TL;DR

This paper introduces a reinforcement learning-based method for optimized bit allocation in image/video coding that enhances task-specific performance while reducing bit-rate, addressing integration challenges with traditional codecs.

Contribution

It is the first to formulate task-driven bit allocation as an MDP and train RL agents for HEVC intra coding, incorporating semantic importance maps for decision-making.

Findings

Achieves 43.1% to 73.2% bit-rate savings compared to HEVC baseline.

Effectively maximizes task-specific fidelity such as classification accuracy.

Demonstrates superior performance through extensive experiments.

Abstract

Rapid growing intelligent applications require optimized bit allocation in image/video coding to support specific task-driven scenarios such as detection, classification, segmentation, etc. Some learning-based frameworks have been proposed for this purpose due to their inherent end-to-end optimization mechanisms. However, it is still quite challenging to integrate these task-driven metrics seamlessly into traditional hybrid coding framework. To the best of our knowledge, this paper is the first work trying to solve this challenge based on reinforcement learning (RL) approach. Specifically, we formulate the bit allocation problem as a Markovian Decision Process (MDP) and train RL agents to automatically decide the quantization parameter (QP) of each coding tree unit (CTU) for HEVC intra coding, according to the task-driven semantic distortion metrics. This bit allocation scheme can maximize the semantic level fidelity of the task, such as classification accuracy, while minimizing the bit-rate. We also employ gradient class activation map (Grad-CAM) and Mask R-CNN tools to extract task-related importance maps to help the agents make decisions. Extensive experimental results demonstrate the superior performance of our approach by achieving 43.1% to 73.2% bit-rate saving over the anchor of HEVC under the equivalent task-related distortions.