Energy-Aware Heterogeneous Federated Learning via Approximate DNN Accelerators

TL;DR

This paper introduces an energy-aware federated learning approach that uses approximate DNN accelerators and compressed arithmetic to significantly reduce energy consumption while maintaining high accuracy across heterogeneous devices.

Contribution

It is the first to incorporate on-device accelerator design with approximate computing and compressed formats in federated learning for energy efficiency.

Findings

Achieves 4x reduction in energy consumption.

Maintains higher accuracy compared to existing methods.

Highlights the importance of memory access costs in energy modeling.

Abstract

In Federated Learning (FL), devices that participate in the training usually have heterogeneous resources, i.e., energy availability. In current deployments of FL, devices that do not fulfill certain hardware requirements are often dropped from the collaborative training. However, dropping devices in FL can degrade training accuracy and introduce bias or unfairness. Several works have tackled this problem on an algorithm level, e.g., by letting constrained devices train a subset of the server neural network (NN) model. However, it has been observed that these techniques are not effective w.r.t. accuracy. Importantly, they make simplistic assumptions about devices' resources via indirect metrics such as multiply accumulate (MAC) operations or peak memory requirements. We observe that memory access costs (that are currently not considered in simplistic metrics) have a significant impact on the energy consumption. In this work, for the first time, we consider on-device accelerator design for FL with heterogeneous devices. We utilize compressed arithmetic formats and approximate computing, targeting to satisfy limited energy budgets. Using a hardware-aware energy model, we observe that, contrary to the state of the art's moderate energy reduction, our technique allows for lowering the energy requirements (by 4x) while maintaining higher accuracy.