Learning with Difference Attention for Visually Grounded Self-supervised Representations

TL;DR

This paper introduces a novel unsupervised visual attention mechanism called visual difference attention (VDA) and a differentiable loss (DiDA) to improve self-supervised learning models' ability to focus on salient regions in complex images, enhancing visual grounding.

Contribution

It proposes VDA as an unsupervised attention method and DiDA loss as a new training objective, significantly improving visual grounding in self-supervised representations.

Findings

VDA highlights salient regions more accurately than existing SSL methods.

DiDA loss improves the visual grounding of SSL models on complex images.

Enhanced representations benefit downstream tasks like segmentation.

Abstract

Recent works in self-supervised learning have shown impressive results on single-object images, but they struggle to perform well on complex multi-object images as evidenced by their poor visual grounding. To demonstrate this concretely, we propose visual difference attention (VDA) to compute visual attention maps in an unsupervised fashion by comparing an image with its salient-regions-masked-out version. We use VDA to derive attention maps for state-of-the art SSL methods and show they do not highlight all salient regions in an image accurately, suggesting their inability to learn strong representations for downstream tasks like segmentation. Motivated by these limitations, we cast VDA as a differentiable operation and propose a new learning objective, Differentiable Difference Attention (DiDA) loss, which leads to substantial improvements in an SSL model's visually grounding to an image's salient regions.