Deriving a Quantitative Relationship Between Resolution and Human Classification Error

TL;DR

This study investigates how image resolution affects human classification accuracy using MNIST data, providing a quantitative heuristic that can inform machine learning benchmarks, data collection, and resource management across various fields.

Contribution

It introduces a quantitative relationship between image resolution and human classification error, aiding in performance prediction and resource optimization.

Findings

Derived a heuristic linking resolution to error rates

Demonstrated the relationship using MNIST dataset

Potential applications in multiple imaging fields

Abstract

For machine learning perception problems, human-level classification performance is used as an estimate of top algorithm performance. Thus, it is important to understand as precisely as possible the factors that impact human-level performance. Knowing this 1) provides a benchmark for model performance, 2) tells a project manager what type of data to obtain for human labelers in order to get accurate labels, and 3) enables ground-truth analysis--largely conducted by humans--to be carried out smoothly. In this empirical study, we explored the relationship between resolution and human classification performance using the MNIST data set down-sampled to various resolutions. The quantitative heuristic we derived could prove useful for predicting machine model performance, predicting data storage requirements, and saving valuable resources in the deployment of machine learning projects. It also has the potential to be used in a wide variety of fields such as remote sensing, medical imaging, scientific imaging, and astronomy.