Parametric Variational Linear Units (PVLUs) in Deep Convolutional Networks

TL;DR

This paper introduces the Parametric Variational Linear Unit (PVLU), a novel activation function that enhances deep CNN performance by reducing error rates through added nonlinearity and fine-tuning capabilities, especially in transfer learning scenarios.

Contribution

The paper proposes PVLU, a new activation function combining ReLU with a trainable sinusoidal component, improving accuracy and transfer learning performance in deep CNNs.

Findings

PVLU reduces error rates by up to 16.3% on CIFAR-100.

PVLU improves transfer learning models like VGG-16 and VGG-19 on CIFAR datasets.

Fine-tuning with PVLU enhances residual network accuracy by over 10%.

Abstract

The Rectified Linear Unit is currently a state-of-the-art activation function in deep convolutional neural networks. To combat ReLU's dying neuron problem, we propose the Parametric Variational Linear Unit (PVLU), which adds a sinusoidal function with trainable coefficients to ReLU. Along with introducing nonlinearity and non-zero gradients across the entire real domain, PVLU acts as a mechanism of fine-tuning when implemented in the context of transfer learning. On a simple, non-transfer sequential CNN, PVLU substitution allowed for relative error decreases of 16.3% and 11.3% (without and with data augmentation) on CIFAR-100. PVLU is also tested on transfer learning models. The VGG-16 and VGG-19 models experience relative error reductions of 9.5% and 10.7% on CIFAR-10, respectively, after the substitution of ReLU with PVLU. When training on Gaussian-filtered CIFAR-10 images, similar improvements are noted for the VGG models. Most notably, fine-tuning using PVLU allows for relative error reductions up to and exceeding 10% for near state-of-the-art residual neural network architectures on the CIFAR datasets.