Neural Frank-Wolfe Policy Optimization for Region-of-Interest Intra-Frame Coding with HEVC/H.265

TL;DR

This paper introduces Neural Frank-Wolfe Policy Optimization, a novel RL framework for ROI intra-frame coding that guarantees convergence and improves bit allocation in HEVC/H.265, outperforming existing methods.

Contribution

It proposes a new RL approach using Frank-Wolfe optimization with dual critics for guaranteed convergence in ROI intra-frame coding.

Findings

Outperforms baseline methods in experimental tests.

Guarantees convergence in the RL-based bit allocation.

Effectively balances rate and distortion in ROI coding.

Abstract

This paper presents a reinforcement learning (RL) framework that utilizes Frank-Wolfe policy optimization to solve Coding-Tree-Unit (CTU) bit allocation for Region-of-Interest (ROI) intra-frame coding. Most previous RL-based methods employ the single-critic design, where the rewards for distortion minimization and rate regularization are weighted by an empirically chosen hyper-parameter. Recently, the dual-critic design is proposed to update the actor by alternating the rate and distortion critics. However, its convergence is not guaranteed. To address these issues, we introduce Neural Frank-Wolfe Policy Optimization (NFWPO) in formulating the CTU-level bit allocation as an action-constrained RL problem. In this new framework, we exploit a rate critic to predict a feasible set of actions. With this feasible set, a distortion critic is invoked to update the actor to maximize the ROI-weighted image quality subject to a rate constraint. Experimental results produced with x265 confirm the superiority of the proposed method to the other baselines.