Assessing and Remedying Coverage for a Given Dataset

TL;DR

This paper presents methods to evaluate and improve dataset coverage across multiple attributes, aiming to reduce bias and vulnerabilities in data-driven decision-making.

Contribution

It introduces efficient techniques for identifying coverage gaps and estimating the minimal additional data needed to address them.

Findings

Effective identification of coverage gaps in datasets.

Quantitative methods to determine minimal data augmentation.

Validated approaches through experiments on real datasets.

Abstract

Data analysis impacts virtually every aspect of our society today. Often, this analysis is performed on an existing dataset, possibly collected through a process that the data scientists had limited control over. The existing data analyzed may not include the complete universe, but it is expected to cover the diversity of items in the universe. Lack of adequate coverage in the dataset can result in undesirable outcomes such as biased decisions and algorithmic racism, as well as creating vulnerabilities such as opening up room for adversarial attacks. In this paper, we assess the coverage of a given dataset over multiple categorical attributes. We first provide efficient techniques for traversing the combinatorial explosion of value combinations to identify any regions of attribute space not adequately covered by the data. Then, we determine the least amount of additional data that must be obtained to resolve this lack of adequate coverage. We confirm the value of our proposal through both theoretical analyses and comprehensive experiments on real data.