A Possible Reason for why Data-Driven Beats Theory-Driven Computer Vision

TL;DR

This paper argues that the success of deep learning in computer vision is partly due to the evolution of empirical practices, which inadvertently favor data-driven models over classical theory-driven algorithms due to dataset biases.

Contribution

It reveals how dataset biases and sensor setting distributions have unintentionally biased comparisons, favoring data-driven models over classical theory-driven approaches.

Findings

Deep learning models outperform classical algorithms on common datasets.

Sensor setting distributions are mismatched with classical algorithms' optimal ranges.

Biases in datasets favor data-driven models in performance comparisons.

Abstract

Why do some continue to wonder about the success and dominance of deep learning methods in computer vision and AI? Is it not enough that these methods provide practical solutions to many problems? Well no, it is not enough, at least for those who feel there should be a science that underpins all of this and that we should have a clear understanding of how this success was achieved. Here, this paper proposes that the dominance we are witnessing would not have been possible by the methods of deep learning alone: the tacit change has been the evolution of empirical practice in computer vision and AI over the past decades. We demonstrate this by examining the distribution of sensor settings in vision datasets and performance of both classic and deep learning algorithms under various camera settings. This reveals a strong mismatch between optimal performance ranges of classical theory-driven algorithms and sensor setting distributions in the common vision datasets, while data-driven models were trained for those datasets. The head-to-head comparisons between data-driven and theory-driven models were therefore unknowingly biased against the theory-driven models.