Assessing Simplification Levels in Neural Networks: The Impact of Hyperparameter Configurations on Complexity and Sensitivity

TL;DR

This paper experimentally examines how hyperparameter choices in neural networks influence their output complexity and sensitivity, providing insights into the relationship between configuration, robustness, and complexity.

Contribution

It offers a systematic analysis of hyperparameter effects on neural network complexity and sensitivity, advancing understanding of their interplay.

Findings

Hyperparameters significantly affect network complexity and robustness.

Activation functions and layer depth influence sensitivity to input perturbations.

Results suggest optimal configurations for balancing complexity and robustness.

Abstract

This paper presents an experimental study focused on understanding the simplification properties of neural networks under different hyperparameter configurations, specifically investigating the effects on Lempel Ziv complexity and sensitivity. By adjusting key hyperparameters such as activation functions, hidden layers, and learning rate, this study evaluates how these parameters impact the complexity of network outputs and their robustness to input perturbations. The experiments conducted using the MNIST dataset aim to provide insights into the relationships between hyperparameters, complexity, and sensitivity, contributing to a deeper theoretical understanding of these concepts in neural networks.