Distributionally Robust Group Backwards Compatibility

TL;DR

This paper addresses backward compatibility issues in machine learning models, especially for underrepresented groups, by applying distributional robustness and minimax fairness techniques, supported by theoretical analysis and experiments on standard datasets.

Contribution

It introduces two novel methods leveraging distributional robustness to improve backward compatibility and fairness across diverse user groups.

Findings

Enhanced model fairness for minority groups

Improved backward compatibility in updated models

Validated methods on CIFAR-10, CelebA, Waterbirds datasets

Abstract

Machine learning models are updated as new data is acquired or new architectures are developed. These updates usually increase model performance, but may introduce backward compatibility errors, where individual users or groups of users see their performance on the updated model adversely affected. This problem can also be present when training datasets do not accurately reflect overall population demographics, with some groups having overall lower participation in the data collection process, posing a significant fairness concern. We analyze how ideas from distributional robustness and minimax fairness can aid backward compatibility in this scenario, and propose two methods to directly address this issue. Our theoretical analysis is backed by experimental results on CIFAR-10, CelebA, and Waterbirds, three standard image classification datasets. Code available at github.com/natalialmg/GroupBC