RCR-AF: Enhancing Model Generalization via Rademacher Complexity Reduction Activation Function

TL;DR

This paper introduces RCR-AF, a novel activation function that reduces model complexity and enhances robustness against adversarial attacks by combining properties of GELU and ReLU with controlled sparsity.

Contribution

The paper proposes RCR-AF, a new activation function that leverages Rademacher complexity theory to improve neural network generalization and adversarial robustness.

Findings

RCR-AF outperforms ReLU, GELU, and Swish in accuracy and robustness.

Theoretical analysis links hyperparameters to model complexity.

Empirical results confirm improved adversarial resilience.

Abstract

Despite their widespread success, deep neural networks remain critically vulnerable to adversarial attacks, posing significant risks in safety-sensitive applications. This paper investigates activation functions as a crucial yet underexplored component for enhancing model robustness. We propose a Rademacher Complexity Reduction Activation Function (RCR-AF), a novel activation function designed to improve both generalization and adversarial resilience. RCR-AF uniquely combines the advantages of GELU (including smoothness, gradient stability, and negative information retention) with ReLU's desirable monotonicity, while simultaneously controlling both model sparsity and capacity through built-in clipping mechanisms governed by two hyperparameters, $\alpha$ and $\gamma$. Our theoretical analysis, grounded in Rademacher complexity, demonstrates that these parameters directly modulate the model's Rademacher complexity, offering a principled approach to enhance robustness. Comprehensive empirical evaluations show that RCR-AF consistently outperforms widely-used alternatives (ReLU, GELU, and Swish) in both clean accuracy under standard training and in adversarial robustness within adversarial training paradigms.