Loss Landscape Analysis for Reliable Quantized ML Models for Scientific Sensing

TL;DR

This paper introduces a method for analyzing the loss landscape of quantized ML models used in scientific sensing, focusing on robustness to noise and perturbations, and enabling more efficient model development.

Contribution

It provides a systematic, empirical approach to assess robustness and trade-offs in quantized ML models without extensive retraining, advancing scientific sensing applications.

Findings

Gently-shaped loss landscapes correlate with robustness.

Quantization impacts model robustness and performance.

Loss landscape analysis reveals non-obvious phenomena.

Abstract

In this paper, we propose a method to perform empirical analysis of the loss landscape of machine learning (ML) models. The method is applied to two ML models for scientific sensing, which necessitates quantization to be deployed and are subject to noise and perturbations due to experimental conditions. Our method allows assessing the robustness of ML models to such effects as a function of quantization precision and under different regularization techniques -- two crucial concerns that remained underexplored so far. By investigating the interplay between performance, efficiency, and robustness by means of loss landscape analysis, we both established a strong correlation between gently-shaped landscapes and robustness to input and weight perturbations and observed other intriguing and non-obvious phenomena. Our method allows a systematic exploration of such trade-offs a priori, i.e., without training and testing multiple models, leading to more efficient development workflows. This work also highlights the importance of incorporating robustness into the Pareto optimization of ML models, enabling more reliable and adaptive scientific sensing systems.