Semi-Periodic Activation for Time Series Classification

TL;DR

This paper introduces LeakySineLU, a new activation function designed for time series neural networks, which outperforms existing activations across numerous benchmark datasets by leveraging properties like periodicity and nonlinearity.

Contribution

The study proposes LeakySineLU, a novel activation function tailored for time series tasks, and demonstrates its superior performance through extensive empirical evaluation.

Findings

LeakySineLU achieves the best average ranking on 112 time series datasets.

Analyzing activation properties reveals the importance of periodicity and nonlinearity.

LeakySineLU outperforms commonly used activation functions in time series classification.

Abstract

This paper investigates the lack of research on activation functions for neural network models in time series tasks. It highlights the need to identify essential properties of these activations to improve their effectiveness in specific domains. To this end, the study comprehensively analyzes properties, such as bounded, monotonic, nonlinearity, and periodicity, for activation in time series neural networks. We propose a new activation that maximizes the coverage of these properties, called LeakySineLU. We empirically evaluate the LeakySineLU against commonly used activations in the literature using 112 benchmark datasets for time series classification, obtaining the best average ranking in all comparative scenarios.