Generalizing Interactive Backpropagating Refinement for Dense Prediction

TL;DR

This paper introduces G-BRS, a generalized backpropagating refinement scheme that enhances dense prediction models across various tasks through both global and localized interactive refinement, improving accuracy with minimal user input.

Contribution

The paper proposes G-BRS layers that extend feature backpropagating refinement to multiple dense prediction tasks, enabling more effective interactive corrections.

Findings

Significant performance improvements on multiple datasets.

Effective global and localized refinement with few user clicks.

Generalizes the f-BRS scheme to diverse dense prediction tasks.

Abstract

As deep neural networks become the state-of-the-art approach in the field of computer vision for dense prediction tasks, many methods have been developed for automatic estimation of the target outputs given the visual inputs. Although the estimation accuracy of the proposed automatic methods continues to improve, interactive refinement is oftentimes necessary for further correction. Recently, feature backpropagating refinement scheme (f-BRS) has been proposed for the task of interactive segmentation, which enables efficient optimization of a small set of auxiliary variables inserted into the pretrained network to produce object segmentation that better aligns with user inputs. However, the proposed auxiliary variables only contain channel-wise scale and bias, limiting the optimization to global refinement only. In this work, in order to generalize backpropagating refinement for a wide range of dense prediction tasks, we introduce a set of G-BRS (Generalized Backpropagating Refinement Scheme) layers that enable both global and localized refinement for the following tasks: interactive segmentation, semantic segmentation, image matting and monocular depth estimation. Experiments on SBD, Cityscapes, Mapillary Vista, Composition-1k and NYU-Depth-V2 show that our method can successfully generalize and significantly improve performance of existing pretrained state-of-the-art models with only a few clicks.