On Point Affiliation in Feature Upsampling

TL;DR

This paper introduces Similarity-Aware Point Affiliation (SAPA), a universal and lightweight upsampling method that improves semantic boundary sharpness and smoothness by leveraging point affiliation based on semantic similarity.

Contribution

The paper proposes a novel point affiliation framework for feature upsampling, leading to the development of SAPA, a universal, similarity-aware kernel method that enhances various dense prediction tasks.

Findings

SAPA outperforms prior upsampling methods across multiple tasks.

SAPA improves boundary sharpness and semantic smoothness.

Extensive experiments validate the effectiveness of SAPA.

Abstract

We introduce the notion of point affiliation into feature upsampling. By abstracting a feature map into non-overlapped semantic clusters formed by points of identical semantic meaning, feature upsampling can be viewed as point affiliation -- designating a semantic cluster for each upsampled point. In the framework of kernel-based dynamic upsampling, we show that an upsampled point can resort to its low-res decoder neighbors and high-res encoder point to reason the affiliation, conditioned on the mutual similarity between them. We therefore present a generic formulation for generating similarity-aware upsampling kernels and prove that such kernels encourage not only semantic smoothness but also boundary sharpness. This formulation constitutes a novel, lightweight, and universal upsampling solution, Similarity-Aware Point Affiliation (SAPA). We show its working mechanism via our preliminary designs with window-shape kernel. After probing the limitations of the designs on object detection, we reveal additional insights for upsampling, leading to SAPA with the dynamic kernel shape. Extensive experiments demonstrate that SAPA outperforms prior upsamplers and invites consistent performance improvements on a number of dense prediction tasks, including semantic segmentation, object detection, instance segmentation, panoptic segmentation, image matting, and depth estimation. Code is made available at: https://github.com/tiny-smart/sapa