Time Regularization in Optimal Time Variable Learning

TL;DR

This paper extends optimal time variable learning in deep neural networks by introducing a regularization term linked to the time horizon and proposes an adaptive pruning method for ResNets to reduce complexity and training time, demonstrated on MNIST datasets.

Contribution

It introduces a novel regularization term for optimal time variable learning and an adaptive pruning approach for ResNets, improving efficiency without sacrificing performance.

Findings

Regularization improves control over the time horizon in DNNs.

Adaptive pruning reduces network complexity and training time.

Effective on MNIST and Fashion MNIST classification tasks.

Abstract

Recently, optimal time variable learning in deep neural networks (DNNs) was introduced in arXiv:2204.08528. In this manuscript we extend the concept by introducing a regularization term that directly relates to the time horizon in discrete dynamical systems. Furthermore, we propose an adaptive pruning approach for Residual Neural Networks (ResNets), which reduces network complexity without compromising expressiveness, while simultaneously decreasing training time. The results are illustrated by applying the proposed concepts to classification tasks on the well known MNIST and Fashion MNIST data sets. Our PyTorch code is available on https://github.com/frederikkoehne/time_variable_learning.