Power Consumption Variation over Activation Functions

TL;DR

This paper investigates how different activation functions impact power consumption in neural networks, highlighting significant hardware performance differences and suggesting ways to reduce energy use.

Contribution

It provides the first comprehensive comparison of power consumption across various activation functions in neural networks.

Findings

Significant power consumption differences between activation functions

Certain activation functions are more energy-efficient on specific hardware

Insights can guide energy-efficient neural network design

Abstract

The power that machine learning models consume when making predictions can be affected by a model's architecture. This paper presents various estimates of power consumption for a range of different activation functions, a core factor in neural network model architecture design. Substantial differences in hardware performance exist between activation functions. This difference informs how power consumption in machine learning models can be reduced.