Semantic segmentation with reward

TL;DR

This paper introduces RSS, a reward-based reinforcement learning approach for semantic segmentation that works with limited labeling, using novel reward mechanisms to improve convergence and outperform weakly supervised methods.

Contribution

The paper presents RSS, the first practical reward-based reinforcement learning method for semantic segmentation using pixel-level and image-level rewards, with novel techniques to enhance training convergence.

Findings

RSS successfully ensures convergence on benchmark datasets.

RSS with image-level rewards outperforms existing weakly supervised methods.

Proposed techniques like PSR and PSD improve reward effectiveness.

Abstract

In real-world scenarios, pixel-level labeling is not always available. Sometimes, we need a semantic segmentation network, and even a visual encoder can have a high compatibility, and can be trained using various types of feedback beyond traditional labels, such as feedback that indicates the quality of the parsing results. To tackle this issue, we proposed RSS (Reward in Semantic Segmentation), the first practical application of reward-based reinforcement learning on pure semantic segmentation offered in two granular levels (pixel-level and image-level). RSS incorporates various novel technologies, such as progressive scale rewards (PSR) and pair-wise spatial difference (PSD), to ensure that the reward facilitates the convergence of the semantic segmentation network, especially under image-level rewards. Experiments and visualizations on benchmark datasets demonstrate that the proposed RSS can successfully ensure the convergence of the semantic segmentation network on two levels of rewards. Additionally, the RSS, which utilizes an image-level reward, outperforms existing weakly supervised methods that also rely solely on image-level signals during training.