Layer Pruning with Consensus: A Triple-Win Solution

TL;DR

This paper introduces a novel layer pruning method called the Consensus criterion that combines multiple similarity metrics to effectively reduce model complexity, improve robustness, and lower energy consumption without significant accuracy loss.

Contribution

It proposes a new multi-metric consensus approach for layer pruning that enhances model efficiency, robustness, and environmental sustainability compared to existing single-criterion methods.

Findings

Up to 78.80% FLOPs reduction with maintained accuracy.

Reduces energy consumption and carbon emissions significantly.

Improves robustness against adversarial attacks by up to 4 percentage points.

Abstract

Layer pruning offers a promising alternative to standard structured pruning, effectively reducing computational costs, latency, and memory footprint. While notable layer-pruning approaches aim to detect unimportant layers for removal, they often rely on single criteria that may not fully capture the complex, underlying properties of layers. We propose a novel approach that combines multiple similarity metrics into a single expressive measure of low-importance layers, called the Consensus criterion. Our technique delivers a triple-win solution: low accuracy drop, high-performance improvement, and increased robustness to adversarial attacks. With up to 78.80% FLOPs reduction and performance on par with state-of-the-art methods across different benchmarks, our approach reduces energy consumption and carbon emissions by up to 66.99% and 68.75%, respectively. Additionally, it avoids shortcut learning and improves robustness by up to 4 percentage points under various adversarial attacks. Overall, the Consensus criterion demonstrates its effectiveness in creating robust, efficient, and environmentally friendly pruned models.