Training Deep Visual Networks Beyond Loss and Accuracy Through a Dynamical Systems Approach

TL;DR

This paper introduces a dynamical systems framework to analyze deep visual network training, focusing on internal layer coordination, stability, and flexibility, providing insights beyond traditional loss and accuracy metrics.

Contribution

It proposes three novel measures—integration, metastability, and stability index—to study training dynamics through layer activations across epochs.

Findings

Integration distinguishes CIFAR-10 from CIFAR-100.

Volatility in stability index signals early convergence.

Relationship between integration and metastability reflects training styles.

Abstract

Deep visual recognition models are usually trained and evaluated using metrics such as loss and accuracy. While these measures show whether a model is improving, they reveal very little about how its internal representations change during training. This paper introduces a complementary way to study that process by examining training through the lens of dynamical systems. Drawing on ideas from signal analysis originally used to study biological neural activity, we define three measures from layer activations collected across training epochs: an integration score that reflects long-range coordination across layers, a metastability score that captures how flexibly the network shifts between more and less synchronised states, and a combined dynamical stability index. We apply this framework to nine combinations of model architecture and dataset, including several ResNet variants, DenseNet-121, MobileNetV2, VGG-16, and a pretrained Vision Transformer on CIFAR-10 and CIFAR-100. The results suggest three main patterns. First, the integration measure consistently distinguishes the easier CIFAR-10 setting from the more difficult CIFAR-100 setting. Second, changes in the volatility of the stability index may provide an early sign of convergence before accuracy fully plateaus. Third, the relationship between integration and metastability appears to reflect different styles of training behaviour. Overall, this study offers an exploratory but promising new way to understand deep visual training beyond loss and accuracy.