Expressive yet Efficient Feature Expansion with Adaptive Cross-Hadamard Products

TL;DR

This paper introduces the ACH module, a learnable, efficient feature expansion operator using adaptive cross-Hadamard products, integrated into Hadamard-Net, achieving state-of-the-art accuracy and efficiency in vision models.

Contribution

The paper proposes a novel adaptive cross-Hadamard operator with learnability and efficiency, integrated into neural architecture search for improved vision model performance.

Findings

Achieves state-of-the-art accuracy and speed on image classification.

Enables efficient feature reuse without extra convolutional parameters.

Establishes Hadamard operations as effective building blocks for vision models.

Abstract

Recent theoretical advances reveal that the Hadamard product induces nonlinear representations and implicit high-dimensional mappings for the field of deep learning, yet their practical deployment in resource-constrained vision models remains largely unexplored. To address this gap, we introduce the Adaptive Cross-Hadamard (ACH) module, a novel operator that embeds learnability through differentiable discrete sampling and dynamic softsign normalization. This facilitates highly efficient feature reuse without incurring additional convolutional parameters, while ensuring stable gradient flow. Integrated into Hadaptive-Net (Hadamard Adaptive Network) via neural architecture search, our approach achieves unprecedented efficiency. Comprehensive experiments demonstrate state-of-the-art accuracy/speed trade-offs on image classification tasks, establishing Hadamard operations as specific building blocks for efficient vision models.