Matching DNN Compression and Cooperative Training with Resources and Data Availability

TL;DR

This paper presents PACT, a novel framework for jointly optimizing DNN compression and training resource allocation, improving efficiency and performance in resource-constrained environments.

Contribution

It introduces a formal multi-dimensional model and an approximate dynamic programming solution for joint decision-making in DNN training and compression.

Findings

PACT outperforms state-of-the-art methods in energy efficiency.

The approach closely matches the optimal solutions in various settings.

The method is polynomial-time and adaptable to different resource scenarios.

Abstract

To make machine learning (ML) sustainable and apt to run on the diverse devices where relevant data is, it is essential to compress ML models as needed, while still meeting the required learning quality and time performance. However, how much and when an ML model should be compressed, and {\em where} its training should be executed, are hard decisions to make, as they depend on the model itself, the resources of the available nodes, and the data such nodes own. Existing studies focus on each of those aspects individually, however, they do not account for how such decisions can be made jointly and adapted to one another. In this work, we model the network system focusing on the training of DNNs, formalize the above multi-dimensional problem, and, given its NP-hardness, formulate an approximate dynamic programming problem that we solve through the PACT algorithmic framework. Importantly, PACT leverages a time-expanded graph representing the learning process, and a data-driven and theoretical approach for the prediction of the loss evolution to be expected as a consequence of training decisions. We prove that PACT's solutions can get as close to the optimum as desired, at the cost of an increased time complexity, and that, in any case, such complexity is polynomial. Numerical results also show that, even under the most disadvantageous settings, PACT outperforms state-of-the-art alternatives and closely matches the optimal energy cost.