Gibbs randomness-compression proposition: An efficient deep learning

TL;DR

This paper introduces a Gibbs entropy-based proposition linking randomness and compression in deep learning, supported by a novel Dual Tomographic Compression framework that enhances training efficiency and neural architecture search.

Contribution

It presents the Gibbs randomness-compression proposition and a Dual Tomographic Compression method, advancing energy-efficient deep learning and neural pruning techniques.

Findings

High correlation between Gibbs entropy and learning performance

DTC accelerates training and supports lottery ticket hypothesis

Random compress-train iterations perform comparably to deterministic methods

Abstract

A proposition that connects randomness and compression is put forward via Gibbs entropy over set of measurement vectors associated with a compression process. The proposition states that a lossy compression process is equivalent to {\it directed randomness} that preserves information content. The proposition originated from the observed behavior in newly proposed {\it Dual Tomographic Compression} (DTC) compress-train framework. This is akin to tomographic reconstruction of layer weight matrices via building compressed sensed projections, via so-called {\it weight rays}. This tomographic approach is applied to previous and next layers in a dual fashion, that triggers neuronal-level pruning. This novel model compress-train scheme appears in iterative fashion and acts as a smart neural architecture search: also called {\it compression aware training}. The experiments demonstrated the utility of this dual-tomography during training: method accelerates and supports lottery ticket hypothesis. However, random compress-train iterations having similar performance demonstrated the connection between randomness and compression from statistical physics perspective, we formulated the so-called {\it Gibbs randomness-compression proposition}, signifying randomness-compression relationship via Gibbs entropy. The proposition is supported with the experimental evidence, resulting in very high correlation between learning performance vs. Gibbs entropy over compression ratios. Practically, the DTC framework provides a promising approach for massively energy- and resource-efficient deep learning training.