ReGLA: Efficient Receptive-Field Modeling with Gated Linear Attention Network

TL;DR

ReGLA introduces a lightweight hybrid Transformer convolutional network with efficient receptive field modeling, achieving high accuracy and low latency on high-resolution images, suitable for real-time visual tasks.

Contribution

The paper presents ReGLA, a novel hybrid network combining efficient convolutions and gated linear attention, with innovations like ELRF and RGMA modules, and a multi-teacher distillation strategy.

Findings

ReGLA-M achieves 80.85% Top-1 accuracy on ImageNet-1K.

ReGLA has 4.98 ms latency at 512px resolution.

ReGLA outperforms similar models in object detection and segmentation tasks.

Abstract

Balancing accuracy and latency on high-resolution images is a critical challenge for lightweight models, particularly for Transformer-based architectures that often suffer from excessive latency. To address this issue, we introduce \textbf{ReGLA}, a series of lightweight hybrid networks, which integrates efficient convolutions for local feature extraction with ReLU-based gated linear attention for global modeling. The design incorporates three key innovations: the Efficient Large Receptive Field (ELRF) module for enhancing convolutional efficiency while preserving a large receptive field; the ReLU Gated Modulated Attention (RGMA) module for maintaining linear complexity while enhancing local feature representation; and a multi-teacher distillation strategy to boost performance on downstream tasks. Extensive experiments validate the superiority of ReGLA; particularly the ReGLA-M achieves \textbf{80.85\%} Top-1 accuracy on ImageNet-1K at $224px$, with only \textbf{4.98 ms} latency at $512px$. Furthermore, ReGLA outperforms similarly scaled iFormer models in downstream tasks, achieving gains of \textbf{3.1\%} AP on COCO object detection and \textbf{3.6\%} mIoU on ADE20K semantic segmentation, establishing it as a state-of-the-art solution for high-resolution visual applications.