How Does Fourier Analysis Network Work? A Mechanism Analysis and a New Dual-Activation Layer Proposal

TL;DR

This paper analyzes the Fourier Analysis Network's mechanism, revealing that sine activation improves training by mitigating vanishing gradients and proposing a Dual-Activation Layer that accelerates convergence across various tasks.

Contribution

It uncovers the specific role of sine activation in FAN, shifts understanding from spectral to training dynamics, and introduces DAL for faster, more effective neural network training.

Findings

Sine activation improves gradient flow near zero, aiding training.

FAN primarily alleviates the dying-ReLU problem.

DAL accelerates convergence and maintains or improves accuracy.

Abstract

Fourier Analysis Network (FAN) was recently proposed as a simple way to improve neural network performance by replacing part of Rectified Linear Unit (ReLU) activations with sine and cosine functions. Although several studies have reported small but consistent gains across tasks, the underlying mechanism behind these improvements has remained unclear. In this work, we show that only the sine activation contributes positively to performance, whereas the cosine activation tends to be detrimental. Our analysis reveals that the improvement is not a consequence of the sine function's periodic nature; instead, it stems from the function's local behavior near x = 0, where its non-zero derivative mitigates the vanishing-gradient problem. We further show that FAN primarily alleviates the dying-ReLU problem, in which a neuron consistently receives negative inputs, produces zero gradients, and stops learning. Although modern ReLU-like activations, such as Leaky ReLU, GELU, and Swish, reduce ReLU's zero-gradient region, they still contain input domains where gradients remain significantly diminished, contributing to slower optimization and hindering rapid convergence. FAN addresses this limitation by introducing a more stable gradient pathway. This analysis shifts the understanding of FAN's benefits from a spectral interpretation to a concrete analysis of training dynamics, leading to the development of the Dual-Activation Layer (DAL), a more efficient convergence accelerator. We evaluate DAL on three tasks: classification of noisy sinusoidal signals versus pure noise, MNIST digit classification, and Electrocardiogram (ECG)-based biometric recognition. In all cases, DAL models converge faster and achieve equal or higher validation accuracy compared to models with conventional activations.