Resource-Constrained On-Device Learning by Dynamic Averaging

TL;DR

This paper presents a communication-efficient, privacy-preserving on-device learning method suitable for low-power processors, significantly reducing energy consumption while maintaining model quality.

Contribution

It introduces a novel approach for on-device learning with integer exponential families that minimizes communication and energy use on low-power hardware.

Findings

Achieves model quality comparable to centralized learning

Reduces communication by an order of magnitude

Significantly lowers overall energy consumption

Abstract

The communication between data-generating devices is partially responsible for a growing portion of the world's power consumption. Thus reducing communication is vital, both, from an economical and an ecological perspective. For machine learning, on-device learning avoids sending raw data, which can reduce communication substantially. Furthermore, not centralizing the data protects privacy-sensitive data. However, most learning algorithms require hardware with high computation power and thus high energy consumption. In contrast, ultra-low-power processors, like FPGAs or micro-controllers, allow for energy-efficient learning of local models. Combined with communication-efficient distributed learning strategies, this reduces the overall energy consumption and enables applications that were yet impossible due to limited energy on local devices. The major challenge is then, that the low-power processors typically only have integer processing capabilities. This paper investigates an approach to communication-efficient on-device learning of integer exponential families that can be executed on low-power processors, is privacy-preserving, and effectively minimizes communication. The empirical evaluation shows that the approach can reach a model quality comparable to a centrally learned regular model with an order of magnitude less communication. Comparing the overall energy consumption, this reduces the required energy for solving the machine learning task by a significant amount.