Measures of Complexity for Large Scale Image Datasets

TL;DR

This paper introduces simple, computationally efficient methods to quantify and visualize the complexity of large-scale image datasets, aiding in dataset comparison and understanding in machine learning.

Contribution

It proposes novel entropy-based complexity metrics and visualization techniques for high-dimensional datasets, applied to autonomous driving image datasets.

Findings

Entropy metrics rank datasets by complexity

Visualizations assist in dataset comparison

Complexity correlates with deep learning difficulty

Abstract

Large scale image datasets are a growing trend in the field of machine learning. However, it is hard to quantitatively understand or specify how various datasets compare to each other - i.e., if one dataset is more complex or harder to ``learn'' with respect to a deep-learning based network. In this work, we build a series of relatively computationally simple methods to measure the complexity of a dataset. Furthermore, we present an approach to demonstrate visualizations of high dimensional data, in order to assist with visual comparison of datasets. We present our analysis using four datasets from the autonomous driving research community - Cityscapes, IDD, BDD and Vistas. Using entropy based metrics, we present a rank-order complexity of these datasets, which we compare with an established rank-order with respect to deep learning.