Pattern-Affinitive Propagation across Depth, Surface Normal and Semantic Segmentation

TL;DR

This paper introduces Pattern-Affinitive Propagation (PAP), a novel framework that enhances joint prediction of depth, surface normal, and semantic segmentation by leveraging pattern affinities across and within tasks, leading to state-of-the-art results.

Contribution

The paper presents a new PAP framework that utilizes cross-task and task-specific propagation based on pattern affinities to improve multi-task learning performance.

Findings

Achieves state-of-the-art results on NYUD-v2, SUN-RGBD, and KITTI datasets.

Demonstrates the effectiveness of cross-task and within-task affinity propagation.

Outperforms existing methods in joint depth, normal, and segmentation prediction.

Abstract

In this paper, we propose a novel Pattern-Affinitive Propagation (PAP) framework to jointly predict depth, surface normal and semantic segmentation. The motivation behind it comes from the statistic observation that pattern-affinitive pairs recur much frequently across different tasks as well as within a task. Thus, we can conduct two types of propagations, cross-task propagation and task-specific propagation, to adaptively diffuse those similar patterns. The former integrates cross-task affinity patterns to adapt to each task therein through the calculation on non-local relationships. Next the latter performs an iterative diffusion in the feature space so that the cross-task affinity patterns can be widely-spread within the task. Accordingly, the learning of each task can be regularized and boosted by the complementary task-level affinities. Extensive experiments demonstrate the effectiveness and the superiority of our method on the joint three tasks. Meanwhile, we achieve the state-of-the-art or competitive results on the three related datasets, NYUD-v2, SUN-RGBD and KITTI.