On the Non-Associativity of Analog Computations

TL;DR

This paper demonstrates that analog computations are non-associative due to input ordering effects, significantly impacting neural network accuracy on resource-constrained devices, highlighting a critical consideration for analog hardware design.

Contribution

It reveals the non-associativity in analog computations caused by input ordering effects and quantifies its impact on neural network accuracy.

Findings

Ordering effects cause substantial accuracy drops.

Analog computations are non-associative despite mathematical associativity.

Neglecting ordering effects leads to significant performance degradation.

Abstract

The energy efficiency of analog forms of computing makes it one of the most promising candidates to deploy resource-hungry machine learning tasks on resource-constrained system such as mobile or embedded devices. However, it is well known that for analog computations the safety net of discretization is missing, thus all analog computations are exposed to a variety of imperfections of corresponding implementations. Examples include non-linearities, saturation effect and various forms of noise. In this work, we observe that the ordering of input operands of an analog operation also has an impact on the output result, which essentially makes analog computations non-associative, even though the underlying operation might be mathematically associative. We conduct a simple test by creating a model of a real analog processor which captures such ordering effects. With this model we assess the importance of ordering by comparing the test accuracy of a neural network for keyword spotting, which is trained based either on an ordered model, on a non-ordered variant, and on real hardware. The results prove the existence of ordering effects as well as their high impact, as neglecting ordering results in substantial accuracy drops.