$\lambda$-GELU: Learning Gating Hardness for Controlled ReLU-ization in Deep Networks

TL;DR

This paper introduces a parameterized GELU variant with a controllable hardness parameter, enabling a smooth transition towards ReLU-like models for better compatibility with existing tools.

Contribution

It proposes a novel hardness-parameterized GELU formulation with a stable learning scheme, allowing controlled ReLU-ization in deep networks.

Findings

Layerwise hardness profiles are structured and robust across models.

Progressive hardening of gates enables ReLU substitution with minimal disruption.

The hardness parameter effectively bridges smooth GELU training and ReLU-compatible models.

Abstract

Gaussian Error Linear Unit (GELU) is a widely used smooth alternative to Rectifier Linear Unit (ReLU), yet many deployment, compression, and analysis toolchains are most naturally expressed for piecewise-linear (ReLU-type) networks. We study a hardness-parameterized formulation of GELU, f(x;{\lambda})=x{\Phi}({\lambda} x), where {\Phi} is the Gaussian CDF and {\lambda} \in [1, infty) controls gate sharpness, with the goal of turning smooth gated training into a controlled path toward ReLU-compatible models. Learning {\lambda} is non-trivial: naive updates yield unstable dynamics and effective gradient attenuation, so we introduce a constrained reparameterization and an optimizer-aware update scheme. Empirically, across a diverse set of model--dataset pairs spanning MLPs, CNNs, and Transformers, we observe structured layerwise hardness profiles and assess their robustness under different initializations. We further study a deterministic ReLU-ization strategy in which the learned gates are progressively hardened toward a principled target, enabling a post-training substitution of {\lambda}-GELU by ReLU with reduced disruption. Overall, {\lambda}-GELU provides a minimal and interpretable knob to profile and control gating hardness, bridging smooth training with ReLU-centric downstream pipelines.