Glimpse-Attend-and-Explore: Self-Attention for Active Visual Exploration

TL;DR

The paper introduces Glimpse-Attend-and-Explore, a self-attention based model for active visual exploration that improves environment understanding across multiple tasks without relying on task-specific uncertainty maps.

Contribution

It presents a novel self-attention approach for active visual exploration applicable to both dense and sparse tasks, outperforming previous methods that depend on uncertainty maps.

Findings

Effective across reconstruction, segmentation, and classification tasks.

Less dependent on dataset bias for exploration.

Self-attention attends to different scene regions during training.

Abstract

Active visual exploration aims to assist an agent with a limited field of view to understand its environment based on partial observations made by choosing the best viewing directions in the scene. Recent methods have tried to address this problem either by using reinforcement learning, which is difficult to train, or by uncertainty maps, which are task-specific and can only be implemented for dense prediction tasks. In this paper, we propose the Glimpse-Attend-and-Explore model which: (a) employs self-attention to guide the visual exploration instead of task-specific uncertainty maps; (b) can be used for both dense and sparse prediction tasks; and (c) uses a contrastive stream to further improve the representations learned. Unlike previous works, we show the application of our model on multiple tasks like reconstruction, segmentation and classification. Our model provides encouraging results while being less dependent on dataset bias in driving the exploration. We further perform an ablation study to investigate the features and attention learned by our model. Finally, we show that our self-attention module learns to attend different regions of the scene by minimizing the loss on the downstream task. Code: https://github.com/soroushseifi/glimpse-attend-explore.