Good Intentions: Adaptive Parameter Management via Intent Signaling

TL;DR

This paper introduces AdaPM, an adaptive parameter management system that uses intent signaling to improve efficiency and eliminate manual tuning in distributed machine learning training.

Contribution

The paper presents a novel intent signaling mechanism and AdaPM, a fully adaptive, zero-tuning parameter manager that outperforms existing methods without manual intervention.

Findings

AdaPM matches or outperforms state-of-the-art managers.

Intent signaling effectively informs parameter management.

Automatic parameter management is feasible and efficient.

Abstract

Parameter management is essential for distributed training of large machine learning (ML) tasks. Some ML tasks are hard to distribute because common approaches to parameter management can be highly inefficient. Advanced parameter management approaches -- such as selective replication or dynamic parameter allocation -- can improve efficiency, but to do so, they typically need to be integrated manually into each task's implementation and they require expensive upfront experimentation to tune correctly. In this work, we explore whether these two problems can be avoided. We first propose a novel intent signaling mechanism that integrates naturally into existing ML stacks and provides the parameter manager with crucial information about parameter accesses. We then describe AdaPM, a fully adaptive, zero-tuning parameter manager based on this mechanism. In contrast to prior systems, this approach separates providing information (simple, done by the task) from exploiting it effectively (hard, done automatically by AdaPM). In our experimental evaluation, AdaPM matched or outperformed state-of-the-art parameter managers out of the box, suggesting that automatic parameter management is possible.