Transition of AI Models in dependence of noise

TL;DR

This paper explores how AI model performance degrades with increasing noise, revealing a universal 'cognition transition' that can be quantitatively characterized using statistical mechanics, aiding in efficient model size estimation.

Contribution

It introduces a scaling approach to understand the cognition transition in AI models under noise, demonstrating universal behavior and providing a method to predict model performance.

Findings

Identifies a power-law scaling of the transition width with model size.

Shows universality in the cognition transition across different models.

Provides a practical tool for estimating necessary model sizes based on scaling laws.

Abstract

We investigate the dependence of the score on noise in the data, and on the network size. As a result, we obtain the so-called "cognition transition" from good performance to zero with increasing noise. The understanding of this transition is of fundamental scientific and practical interest. We use concepts from statistical mechanics to understand how a changing finite size of models affects the cognition ability under the presence or corrupted data. On one hand, we study if there is a universal aspect in the transition to several models, on the other hand we go into detail how the approach of the cognition transition point can be captured quantitatively. Therefore, we use the so-called scaling approach from statistical mechanics and find a power-law behaviour of the transition width with increasing model size. Since our study is aimed at universal aspects we use well-know models and data for image classification. That way we avoid uncertainties in data handling or model setup. The practical implication of our results is a tool to estimate model sizes for a certain "universality class" of models, without the need to investigate large sizes, just by extrapolating the scaling results. In turn, that allows for cost reduction in hyperparameter studies. Here, we present first results on a concrete setup; we think that the understanding the mechanics of large system sizes is of fundamental interest for a further exploration of even larger models.