Learning Task-Specific Generalized Convolutions in the Permutohedral Lattice

TL;DR

This paper introduces a novel task-specific generalized convolution layer based on the permutohedral lattice, which improves dense prediction tasks by reducing boundary artifacts and enhancing accuracy through learned feature representations.

Contribution

It extends permutohedral lattice convolutions by learning feature representations in a task-specific manner, enabling non-local operations and improved boundary handling.

Findings

Reduces boundary artifacts in dense prediction tasks.

Improves accuracy in optical flow and semantic segmentation.

Demonstrates versatility across different joint upsampling tasks.

Abstract

Dense prediction tasks typically employ encoder-decoder architectures, but the prevalent convolutions in the decoder are not image-adaptive and can lead to boundary artifacts. Different generalized convolution operations have been introduced to counteract this. We go beyond these by leveraging guidance data to redefine their inherent notion of proximity. Our proposed network layer builds on the permutohedral lattice, which performs sparse convolutions in a high-dimensional space allowing for powerful non-local operations despite small filters. Multiple features with different characteristics span this permutohedral space. In contrast to prior work, we learn these features in a task-specific manner by generalizing the basic permutohedral operations to learnt feature representations. As the resulting objective is complex, a carefully designed framework and learning procedure are introduced, yielding rich feature embeddings in practice. We demonstrate the general applicability of our approach in different joint upsampling tasks. When adding our network layer to state-of-the-art networks for optical flow and semantic segmentation, boundary artifacts are removed and the accuracy is improved.