Variational Structured Attention Networks for Deep Visual Representation Learning

TL;DR

This paper introduces VISTA-Net, a unified probabilistic framework that jointly learns structured spatial and channel attention for enhanced deep visual representation, significantly improving performance on various dense prediction tasks.

Contribution

It proposes a novel end-to-end trainable model that structures and models interactions between spatial and channel attentions within a probabilistic framework.

Findings

Outperforms state-of-the-art on six large-scale datasets

Effective joint learning of spatial and channel attentions

Improves accuracy in dense visual prediction tasks

Abstract

Convolutional neural networks have enabled major progresses in addressing pixel-level prediction tasks such as semantic segmentation, depth estimation, surface normal prediction and so on, benefiting from their powerful capabilities in visual representation learning. Typically, state of the art models integrate attention mechanisms for improved deep feature representations. Recently, some works have demonstrated the significance of learning and combining both spatial- and channelwise attentions for deep feature refinement. In this paper, weaim at effectively boosting previous approaches and propose a unified deep framework to jointly learn both spatial attention maps and channel attention vectors in a principled manner so as to structure the resulting attention tensors and model interactions between these two types of attentions. Specifically, we integrate the estimation and the interaction of the attentions within a probabilistic representation learning framework, leading to VarIational STructured Attention networks (VISTA-Net). We implement the inference rules within the neural network, thus allowing for end-to-end learning of the probabilistic and the CNN frontend parameters. As demonstrated by our extensive empirical evaluation on six large-scale datasets for dense visual prediction, VISTA-Net outperforms the state-of-the-art in multiple continuous and discrete prediction tasks, thus confirming the benefit of the proposed approach in joint structured spatial-channel attention estimation for deep representation learning. The code is available at https://github.com/ygjwd12345/VISTA-Net.