DeconDTN-Toolkit: A Library for Evaluation and Enhancement of Robustness to Provenance Shift

TL;DR

This paper introduces DeconDTN-Toolkit, a comprehensive library for evaluating and improving model robustness against provenance shifts, which are changes in data source relationships affecting deployment performance.

Contribution

It establishes a formal connection between provenance shift and invariant learning, and provides a toolkit for simulation, evaluation, and mitigation of provenance-related distribution shifts.

Findings

Empirical Risk Minimization is vulnerable under provenance shift.

The toolkit enables simulation of provenance shifts of varying degrees.

A new robustness indicator improves out-of-distribution performance assessment.

Abstract

Despite the burgeoning body of work on distribution shifts, provenance shift-where the relationship between data source and label changes at deployment-remains poorly understood and under-addressed. In this paper, we establish a formal connection between provenance shift, counterfactual invariance, and invariant learning to derive a learning objective for robustness. We then introduce \textsc{DeconDTN-Toolkit}, a specialized evaluation and remediation suite designed to simulate provenance shifts of varying degrees while maintaining the training protocol and the infrastructure of existing benchmarks. We reveal the vulnerability of Empirical Risk Minimization under provenance shift, introduce a robust out-of-distribution performance indicator, and conduct a comprehensive evaluation on existing algorithms. Our work provides both the theoretical grounding and the practical tools necessary to characterize the problem of confounding by provenance, and implementations of methods to mitigate it.