Blurring the Line Between Structure and Learning to Optimize and Adapt Receptive Fields

TL;DR

This paper introduces a semi-structured filter composition that adaptively optimizes receptive fields for improved semantic segmentation, combining expressiveness with efficiency and dynamic adaptation to local scale variations.

Contribution

It proposes a novel semi-structured filtering approach that optimizes receptive field size and shape end-to-end, enabling dynamic adaptation for better segmentation performance.

Findings

Improves Cityscapes segmentation accuracy by 1-2 points with static filters.

Further enhances accuracy up to 10 points with dynamic Gaussian structure.

Achieves efficiency comparable to deformation methods while maintaining high accuracy.

Abstract

The visual world is vast and varied, but its variations divide into structured and unstructured factors. We compose free-form filters and structured Gaussian filters, optimized end-to-end, to factorize deep representations and learn both local features and their degree of locality. Our semi-structured composition is strictly more expressive than free-form filtering, and changes in its structured parameters would require changes in free-form architecture. In effect this optimizes over receptive field size and shape, tuning locality to the data and task. Dynamic inference, in which the Gaussian structure varies with the input, adapts receptive field size to compensate for local scale variation. Optimizing receptive field size improves semantic segmentation accuracy on Cityscapes by 1-2 points for strong dilated and skip architectures and by up to 10 points for suboptimal designs. Adapting receptive fields by dynamic Gaussian structure further improves results, equaling the accuracy of free-form deformation while improving efficiency.