UltraMNIST Classification: A Benchmark to Train CNNs for Very Large Images

TL;DR

This paper introduces UltraMNIST, a benchmark dataset designed to facilitate the development of CNN models capable of effectively processing very large images, addressing challenges like memory constraints and multi-scale feature extraction.

Contribution

The paper presents the UltraMNIST dataset and two benchmark variants, enabling research on CNN training for large images under resource constraints and with improved multi-scale feature handling.

Findings

Baseline models show performance degradation with reduced resolution.

Pretrained backbones improve classification accuracy.

Budget-aware models can operate within limited GPU memory.

Abstract

Convolutional neural network (CNN) approaches available in the current literature are designed to work primarily with low-resolution images. When applied on very large images, challenges related to GPU memory, smaller receptive field than needed for semantic correspondence and the need to incorporate multi-scale features arise. The resolution of input images can be reduced, however, with significant loss of critical information. Based on the outlined issues, we introduce a novel research problem of training CNN models for very large images, and present 'UltraMNIST dataset', a simple yet representative benchmark dataset for this task. UltraMNIST has been designed using the popular MNIST digits with additional levels of complexity added to replicate well the challenges of real-world problems. We present two variants of the problem: 'UltraMNIST classification' and 'Budget-aware UltraMNIST classification'. The standard UltraMNIST classification benchmark is intended to facilitate the development of novel CNN training methods that make the effective use of the best available GPU resources. The budget-aware variant is intended to promote development of methods that work under constrained GPU memory. For the development of competitive solutions, we present several baseline models for the standard benchmark and its budget-aware variant. We study the effect of reducing resolution on the performance and present results for baseline models involving pretrained backbones from among the popular state-of-the-art models. Finally, with the presented benchmark dataset and the baselines, we hope to pave the ground for a new generation of CNN methods suitable for handling large images in an efficient and resource-light manner.