The Simpler The Better: An Entropy-Based Importance Metric To Reduce Neural Networks' Depth

TL;DR

This paper introduces EASIER, an entropy-based importance metric that effectively reduces the depth of over-parametrized neural networks, leading to more efficient models without sacrificing performance, especially useful for simpler downstream tasks.

Contribution

The paper presents a novel entropy-based importance metric (EASIER) for neural network depth reduction, improving efficiency while maintaining accuracy.

Findings

EASIER effectively reduces network depth in image classification tasks.

The method decreases computational costs of large models.

EASIER maintains performance despite network simplification.

Abstract

While deep neural networks are highly effective at solving complex tasks, large pre-trained models are commonly employed even to solve consistently simpler downstream tasks, which do not necessarily require a large model's complexity. Motivated by the awareness of the ever-growing AI environmental impact, we propose an efficiency strategy that leverages prior knowledge transferred by large models. Simple but effective, we propose a method relying on an Entropy-bASed Importance mEtRic (EASIER) to reduce the depth of over-parametrized deep neural networks, which alleviates their computational burden. We assess the effectiveness of our method on traditional image classification setups. Our code is available at https://github.com/VGCQ/EASIER.