Deep Nets: What have they ever done for Vision?

TL;DR

This opinion paper discusses the strengths and limitations of Deep Nets in vision, emphasizing their successes in specific tasks but highlighting fundamental challenges in achieving human-like understanding due to the complexity of natural images.

Contribution

The paper critically evaluates Deep Nets' capabilities in vision, arguing that current methods are insufficient for overcoming the combinatorial complexity of natural scenes and calling for new benchmarking approaches.

Findings

Deep Nets excel at specific visual benchmarks.

They lack the flexibility and generalization of human vision.

Current methods are unlikely to solve the fundamental combinatorial problem.

Abstract

This is an opinion paper about the strengths and weaknesses of Deep Nets for vision. They are at the heart of the enormous recent progress in artificial intelligence and are of growing importance in cognitive science and neuroscience. They have had many successes but also have several limitations and there is limited understanding of their inner workings. At present Deep Nets perform very well on specific visual tasks with benchmark datasets but they are much less general purpose, flexible, and adaptive than the human visual system. We argue that Deep Nets in their current form are unlikely to be able to overcome the fundamental problem of computer vision, namely how to deal with the combinatorial explosion, caused by the enormous complexity of natural images, and obtain the rich understanding of visual scenes that the human visual achieves. We argue that this combinatorial explosion takes us into a regime where "big data is not enough" and where we need to rethink our methods for benchmarking performance and evaluating vision algorithms. We stress that, as vision algorithms are increasingly used in real world applications, that performance evaluation is not merely an academic exercise but has important consequences in the real world. It is impractical to review the entire Deep Net literature so we restrict ourselves to a limited range of topics and references which are intended as entry points into the literature. The views expressed in this paper are our own and do not necessarily represent those of anybody else in the computer vision community.