Gaussian Error Linear Units (GELUs)

TL;DR

The paper introduces GELU, a new neural network activation function based on Gaussian error, which outperforms ReLU and ELU across various tasks in vision, language, and speech domains.

Contribution

It proposes the GELU activation function, combining Gaussian error properties with neural network nonlinearities, and demonstrates its superior performance empirically.

Findings

GELU outperforms ReLU and ELU in multiple tasks

GELU improves model accuracy across vision, NLP, and speech

Empirical results show consistent performance gains

Abstract

We propose the Gaussian Error Linear Unit (GELU), a high-performing neural network activation function. The GELU activation function is $x\Phi(x)$, where $\Phi(x)$ the standard Gaussian cumulative distribution function. The GELU nonlinearity weights inputs by their value, rather than gates inputs by their sign as in ReLUs ($x\mathbf{1}_{x>0}$). We perform an empirical evaluation of the GELU nonlinearity against the ReLU and ELU activations and find performance improvements across all considered computer vision, natural language processing, and speech tasks.