Using Topological Framework for the Design of Activation Function and Model Pruning in Deep Neural Networks

TL;DR

This paper introduces a topological approach to designing activation functions that accelerate training convergence and a pruning method that reduces model complexity by removing filters associated with high topological complexity, demonstrated on various datasets.

Contribution

It proposes a novel topologically-informed activation function for faster training and a filter pruning technique based on Betti numbers to simplify models without accuracy loss.

Findings

Activation function reduces training epochs by 1.5 to 2 times.

Pruning filters with high Betti numbers maintains accuracy.

Faster inference and smaller models achieved.

Abstract

Success of deep neural networks in diverse tasks across domains of computer vision, speech recognition and natural language processing, has necessitated understanding the dynamics of training process and also working of trained models. Two independent contributions of this paper are 1) Novel activation function for faster training convergence 2) Systematic pruning of filters of models trained irrespective of activation function. We analyze the topological transformation of the space of training samples as it gets transformed by each successive layer during training, by changing the activation function. The impact of changing activation function on the convergence during training is reported for the task of binary classification. A novel activation function aimed at faster convergence for classification tasks is proposed. Here, Betti numbers are used to quantify topological complexity of data. Results of experiments on popular synthetic binary classification datasets with large Betti numbers(>150) using MLPs are reported. Results show that the proposed activation function results in faster convergence requiring fewer epochs by a factor of 1.5 to 2, since Betti numbers reduce faster across layers with the proposed activation function. The proposed methodology was verified on benchmark image datasets: fashion MNIST, CIFAR-10 and cat-vs-dog images, using CNNs. Based on empirical results, we propose a novel method for pruning a trained model. The trained model was pruned by eliminating filters that transform data to a topological space with large Betti numbers. All filters with Betti numbers greater than 300 were removed from each layer without significant reduction in accuracy. This resulted in faster prediction time and reduced memory size of the model.