Heterogeneous Grid Convolution for Adaptive, Efficient, and Controllable Computation

TL;DR

This paper introduces a heterogeneous grid convolution method that creates adaptive graph-based image representations, leading to more efficient and controllable neural networks with strong performance on multiple image understanding tasks.

Contribution

It presents a novel differentiable clustering-based graph construction and a direction-aware graph convolution, enabling adaptive and efficient convolutional architectures.

Findings

Outperforms baseline with over 90% FLOP reduction in semantic segmentation

Achieves state-of-the-art results in road extraction

Effective on three out of four evaluated tasks

Abstract

This paper proposes a novel heterogeneous grid convolution that builds a graph-based image representation by exploiting heterogeneity in the image content, enabling adaptive, efficient, and controllable computations in a convolutional architecture. More concretely, the approach builds a data-adaptive graph structure from a convolutional layer by a differentiable clustering method, pools features to the graph, performs a novel direction-aware graph convolution, and unpool features back to the convolutional layer. By using the developed module, the paper proposes heterogeneous grid convolutional networks, highly efficient yet strong extension of existing architectures. We have evaluated the proposed approach on four image understanding tasks, semantic segmentation, object localization, road extraction, and salient object detection. The proposed method is effective on three of the four tasks. Especially, the method outperforms a strong baseline with more than 90% reduction in floating-point operations for semantic segmentation, and achieves the state-of-the-art result for road extraction. We will share our code, model, and data.