Machine Learning vs Deep Learning: The Generalization Problem

TL;DR

This paper compares traditional machine learning and deep learning models in their ability to generalize beyond training data, highlighting deep learning's superior extrapolation capabilities through empirical analysis.

Contribution

It provides an empirical comparison of ML and DL models' extrapolation abilities using a specific function, emphasizing deep learning's potential for better generalization beyond training data.

Findings

Deep learning models can better extrapolate beyond training data.

ML models struggle with predictions outside their training domain.

Results highlight the importance of model choice for real-world generalization.

Abstract

The capacity to generalize beyond the range of training data is a pivotal challenge, often synonymous with a model's utility and robustness. This study investigates the comparative abilities of traditional machine learning (ML) models and deep learning (DL) algorithms in terms of extrapolation -- a more challenging aspect of generalization because it requires the model to make inferences about data points that lie outside the domain it has been trained on. We present an empirical analysis where both ML and DL models are trained on an exponentially growing function and then tested on values outside the training domain. The choice of this function allows us to distinctly showcase the divergence in performance when models are required to predict beyond the scope of their training data. Our findings suggest that deep learning models possess inherent capabilities to generalize beyond the training scope, an essential feature for real-world applications where data is often incomplete or extends beyond the observed range. This paper argues for a nuanced understanding of the structural differences between ML and DL models, with an emphasis on the implications for both theoretical research and practical deployment.