SG-Blend: Learning an Interpolation Between Improved Swish and GELU for Robust Neural Representations

TL;DR

SG-Blend is a new adaptive activation function that combines SSwish and GELU to improve neural network robustness and performance across various tasks with minimal computational cost.

Contribution

It introduces SG-Blend, a learnable interpolation of SSwish and GELU, enhancing model robustness and performance across multiple domains.

Findings

SG-Blend outperforms baseline activation functions in NLP and vision tasks.

It achieves these improvements with negligible computational overhead.

Empirical results demonstrate consistent performance gains across diverse architectures.

Abstract

The design of activation functions remains a pivotal component in optimizing deep neural networks. While prevailing choices like Swish and GELU demonstrate considerable efficacy, they often exhibit domain-specific optima. This work introduces SG-Blend, a novel activation function that blends our proposed SSwish, a first-order symmetric variant of Swish and the established GELU through dynamic interpolation. By adaptively blending these constituent functions via learnable parameters, SG-Blend aims to harness their complementary strengths: SSwish's controlled non-monotonicity and symmetry, and GELU's smooth, probabilistic profile, to achieve a more universally robust balance between model expressivity and gradient stability. We conduct comprehensive empirical evaluations across diverse modalities and architectures, showing performance improvements across all considered natural language and computer vision tasks and models. These results, achieved with negligible computational overhead, underscore SG-Blend's potential as a versatile, drop-in replacement that consistently outperforms strong contemporary baselines. The code is available at https://anonymous.4open.science/r/SGBlend-6CBC.