To update or not to update? Neurons at equilibrium in deep models

TL;DR

This paper introduces a neuron-centric approach called neuronal equilibrium (NEq) that determines when to update neurons in deep models, aiming to improve training efficiency and understanding of neuron behavior.

Contribution

It shifts focus from individual parameters to neuron behavior, proposing NEq to selectively update neurons based on their equilibrium state, enhancing training strategies.

Findings

NEq effectively identifies neurons at equilibrium to halt updates.

The approach adapts to different learning strategies and tasks.

Neuronal equilibrium varies with the learning setup.

Abstract

Recent advances in deep learning optimization showed that, with some a-posteriori information on fully-trained models, it is possible to match the same performance by simply training a subset of their parameters. Such a discovery has a broad impact from theory to applications, driving the research towards methods to identify the minimum subset of parameters to train without look-ahead information exploitation. However, the methods proposed do not match the state-of-the-art performance, and rely on unstructured sparsely connected models. In this work we shift our focus from the single parameters to the behavior of the whole neuron, exploiting the concept of neuronal equilibrium (NEq). When a neuron is in a configuration at equilibrium (meaning that it has learned a specific input-output relationship), we can halt its update; on the contrary, when a neuron is at non-equilibrium, we let its state evolve towards an equilibrium state, updating its parameters. The proposed approach has been tested on different state-of-the-art learning strategies and tasks, validating NEq and observing that the neuronal equilibrium depends on the specific learning setup.