Linear Attention Mechanism: An Efficient Attention for Semantic Segmentation

TL;DR

This paper introduces a Linear Attention Mechanism that approximates dot-product attention with significantly reduced memory and computational costs, enhancing the efficiency of semantic segmentation models.

Contribution

The paper presents a novel linear attention mechanism that is more efficient and flexible, enabling better integration with neural networks for semantic segmentation.

Findings

Demonstrated effectiveness on semantic segmentation tasks

Reduced memory and computational costs compared to traditional attention

Code implementation available online

Abstract

In this paper, to remedy this deficiency, we propose a Linear Attention Mechanism which is approximate to dot-product attention with much less memory and computational costs. The efficient design makes the incorporation between attention mechanisms and neural networks more flexible and versatile. Experiments conducted on semantic segmentation demonstrated the effectiveness of linear attention mechanism. Code is available at https://github.com/lironui/Linear-Attention-Mechanism.