Activation-Free Backbones for Image Recognition: Polynomial Alternatives within MetaFormer-Style Vision Models

TL;DR

This paper introduces activation-free polynomial modules for vision backbones within MetaFormer architectures, achieving comparable or superior performance to traditional nonlinear models across multiple vision tasks.

Contribution

It proposes polynomial alternatives to nonlinear primitives in vision backbones, enabling activation-free models that outperform prior polynomial networks at lower computational costs.

Findings

PolyNeXt models match or surpass activation-based models on ImageNet and ADE20K.

Polynomial modules outperform complex architectures with less computation.

Activation-free design maintains high performance without nonlinearities.

Abstract

Modern vision backbones treat pointwise activations (e.g., ReLU, GELU) and exponential softmax as essential sources of nonlinearity, but we demonstrate they are not required within MetaFormer-style vision backbones. We design activation-free polynomial alternatives for three core primitives (MLPs, convolutions, and attention), where Hadamard products replace standard nonlinearities to yield polynomial functions of the input. These modules integrate seamlessly into existing architectures: instantiated within MetaFormer, a modular framework for vision backbones, our PolyNeXt models match or exceed activation-based counterparts across model scales on ImageNet classification, ADE20K semantic segmentation, and out-of-distribution robustness. We also substantially outperform prior polynomial networks at reduced computational cost, showing that polynomial variants of standard modules beat complex custom architectures.