Impact of Learning Rate on Noise Resistant Property of Deep Learning Models

TL;DR

This paper investigates how the learning rate affects the noise resistance of deep learning models in analog computation, identifying optimal values for robustness and performance balance, supported by theoretical analysis.

Contribution

It provides new insights into the role of learning rate in enhancing noise resistance of deep learning models for analog hardware.

Findings

Existence of a learning rate 'sweet spot' balancing performance and noise resistance

Optimal learning rates improve model robustness against analog noise

Theoretical explanation of the observed relationship

Abstract

The interest in analog computation has grown tremendously in recent years due to its fast computation speed and excellent energy efficiency, which is very important for edge and IoT devices in the sub-watt power envelope for deep learning inferencing. However, significant performance degradation suffered by deep learning models due to the inherent noise present in the analog computation can limit their use in mission-critical applications. Hence, there is a need to understand the impact of critical model hyperparameters choice on the resulting model noise-resistant property. This need is critical as the insight obtained can be used to design deep learning models that are robust to analog noise. In this paper, the impact of the learning rate, a critical design choice, on the noise-resistant property is investigated. The study is achieved by first training deep learning models using different learning rates. Thereafter, the models are injected with analog noise and the noise-resistant property of the resulting models is examined by measuring the performance degradation due to the analog noise. The results showed there exists a sweet spot of learning rate values that achieves a good balance between model prediction performance and model noise-resistant property. Furthermore, the theoretical justification of the observed phenomenon is provided.