INSIDE: Steering Spatial Attention with Non-Imaging Information in CNNs

TL;DR

This paper introduces INSIDE, a novel spatial attention mechanism that incorporates non-imaging information into CNNs for improved segmentation, enabling spatial localization conditioned on factors like lesion location or cardiac phase.

Contribution

We propose INSIDE, a differentiable, end-to-end trainable spatial conditioning method using feature-wise attention with parametrized functions, enhancing CNN segmentation with non-imaging data.

Findings

Improved segmentation accuracy on CLEVR-Seg and ACDC datasets.

Effective spatial localization conditioned on non-imaging factors.

No additional supervision required for training.

Abstract

We consider the problem of integrating non-imaging information into segmentation networks to improve performance. Conditioning layers such as FiLM provide the means to selectively amplify or suppress the contribution of different feature maps in a linear fashion. However, spatial dependency is difficult to learn within a convolutional paradigm. In this paper, we propose a mechanism to allow for spatial localisation conditioned on non-imaging information, using a feature-wise attention mechanism comprising a differentiable parametrised function (e.g. Gaussian), prior to applying the feature-wise modulation. We name our method INstance modulation with SpatIal DEpendency (INSIDE). The conditioning information might comprise any factors that relate to spatial or spatio-temporal information such as lesion location, size, and cardiac cycle phase. Our method can be trained end-to-end and does not require additional supervision. We evaluate the method on two datasets: a new CLEVR-Seg dataset where we segment objects based on location, and the ACDC dataset conditioned on cardiac phase and slice location within the volume. Code and the CLEVR-Seg dataset are available at https://github.com/jacenkow/inside.