Topology-Aware Activation Functions in Neural Networks

TL;DR

This paper introduces topology-aware activation functions, $ ext{SmoothSplit}$ and $ ext{ParametricSplit}$, that improve neural networks' ability to manipulate data topology, especially in low-dimensional layers, leading to better performance.

Contribution

It proposes novel topology-aware activation functions that enable neural networks to better manipulate data topology during training, addressing limitations of traditional activations.

Findings

$ ext{ParametricSplit}$ outperforms traditional activations in low-dimensional data scenarios.

The proposed functions effectively transform complex data manifolds.

Experimental results validate the advantages of topology-aware activations.

Abstract

This study explores novel activation functions that enhance the ability of neural networks to manipulate data topology during training. Building on the limitations of traditional activation functions like $\mathrm{ReLU}$, we propose $\mathrm{SmoothSplit}$ and $\mathrm{ParametricSplit}$, which introduce topology "cutting" capabilities. These functions enable networks to transform complex data manifolds effectively, improving performance in scenarios with low-dimensional layers. Through experiments on synthetic and real-world datasets, we demonstrate that $\mathrm{ParametricSplit}$ outperforms traditional activations in low-dimensional settings while maintaining competitive performance in higher-dimensional ones. Our findings highlight the potential of topology-aware activation functions in advancing neural network architectures. The code is available via https://github.com/Snopoff/Topology-Aware-Activations.