RecConv: Efficient Recursive Convolutions for Multi-Frequency Representations

TL;DR

RecConv introduces a recursive convolution strategy that efficiently expands the receptive field with minimal parameter increase and constant FLOPs, enabling more efficient vision transformer models.

Contribution

It proposes RecConv, a recursive decomposition method for multi-frequency representations that maintains constant FLOPs while significantly increasing the receptive field.

Findings

RecConv achieves a linear parameter growth with decomposition levels.

RecConv maintains constant FLOPs regardless of receptive field expansion.

RecNeXt-M3 outperforms comparable models with similar FLOPs on COCO.

Abstract

Recent advances in vision transformers (ViTs) have demonstrated the advantage of global modeling capabilities, prompting widespread integration of large-kernel convolutions for enlarging the effective receptive field (ERF). However, the quadratic scaling of parameter count and computational complexity (FLOPs) with respect to kernel size poses significant efficiency and optimization challenges. This paper introduces RecConv, a recursive decomposition strategy that efficiently constructs multi-frequency representations using small-kernel convolutions. RecConv establishes a linear relationship between parameter growth and decomposing levels which determines the effective receptive field $k\times 2^\ell$ for a base kernel $k$ and $\ell$ levels of decomposition, while maintaining constant FLOPs regardless of the ERF expansion. Specifically, RecConv achieves a parameter expansion of only $\ell+2$ times and a maximum FLOPs increase of $5/3$ times, compared to the exponential growth ($4^\ell$) of standard and depthwise convolutions. RecNeXt-M3 outperforms RepViT-M1.1 by 1.9 $AP^{box}$ on COCO with similar FLOPs. This innovation provides a promising avenue towards designing efficient and compact networks across various modalities. Codes and models can be found at https://github.com/suous/RecNeXt.