Learning 1D Causal Visual Representation with De-focus Attention Networks

TL;DR

This paper introduces De-focus Attention Networks, a novel approach employing learnable filters and training strategies to enable 1D causal visual representations to match the performance of traditional 2D non-causal models across various vision tasks.

Contribution

It proposes a new attention mechanism with learnable bandpass filters and training techniques to address over-focus in 1D causal vision models, enabling effective multi-task visual understanding.

Findings

1D causal visual models can perform comparably to 2D non-causal models.

De-focus Attention Networks improve attention diversity and model optimization.

The approach is validated across perception, prediction, and multi-modal tasks.

Abstract

Modality differences have led to the development of heterogeneous architectures for vision and language models. While images typically require 2D non-causal modeling, texts utilize 1D causal modeling. This distinction poses significant challenges in constructing unified multi-modal models. This paper explores the feasibility of representing images using 1D causal modeling. We identify an "over-focus" issue in existing 1D causal vision models, where attention overly concentrates on a small proportion of visual tokens. The issue of "over-focus" hinders the model's ability to extract diverse visual features and to receive effective gradients for optimization. To address this, we propose De-focus Attention Networks, which employ learnable bandpass filters to create varied attention patterns. During training, large and scheduled drop path rates, and an auxiliary loss on globally pooled features for global understanding tasks are introduced. These two strategies encourage the model to attend to a broader range of tokens and enhance network optimization. Extensive experiments validate the efficacy of our approach, demonstrating that 1D causal visual representation can perform comparably to 2D non-causal representation in tasks such as global perception, dense prediction, and multi-modal understanding. Code is released at https://github.com/OpenGVLab/De-focus-Attention-Networks.