On Data-centric Myths

TL;DR

This paper critically examines common beliefs about data quality in machine learning, showing that some intuitions are misleading and advocating for a more nuanced, data-aware theoretical framework.

Contribution

It challenges existing data-centric myths, providing empirical evidence that questions traditional assumptions about data minimization and distribution preservation.

Findings

Data dimension should not necessarily be minimized.

Preserving data distribution is not always essential when manipulating data.

Existing intuitions about data quality can be misleading.

Abstract

The community lacks theory-informed guidelines for building good data sets. We analyse theoretical directions relating to what aspects of the data matter and conclude that the intuitions derived from the existing literature are incorrect and misleading. Using empirical counter-examples, we show that 1) data dimension should not necessarily be minimised and 2) when manipulating data, preserving the distribution is inessential. This calls for a more data-aware theoretical understanding. Although not explored in this work, we propose the study of the impact of data modification on learned representations as a promising research direction.