Differentially Private High-dimensional Variable Selection via Integer Programming

TL;DR

This paper introduces differentially private estimators for high-dimensional sparse variable selection using advanced mixed integer programming techniques, providing theoretical guarantees and demonstrating superior empirical support recovery in large-scale problems.

Contribution

The paper develops novel pure differentially private estimators for sparse variable selection leveraging modern MIP methods, with theoretical guarantees and improved empirical performance.

Findings

Achieves state-of-the-art support recovery in high-dimensional settings.

Outperforms competing algorithms with up to 10,000 variables.

Provides theoretical support guarantees for the proposed estimators.

Abstract

Sparse variable selection improves interpretability and generalization in high-dimensional learning by selecting a small subset of informative features. Recent advances in Mixed Integer Programming (MIP) have enabled solving large-scale non-private sparse regression - known as Best Subset Selection (BSS) - with millions of variables in minutes. However, extending these algorithmic advances to the setting of Differential Privacy (DP) has remained largely unexplored. In this paper, we introduce two new pure differentially private estimators for sparse variable selection, levering modern MIP techniques. Our framework is general and applies broadly to problems like sparse regression or classification, and we provide theoretical support recovery guarantees in the case of BSS. Inspired by the exponential mechanism, we develop structured sampling procedures that efficiently explore the non-convex objective landscape, avoiding the exhaustive combinatorial search in the exponential mechanism. We complement our theoretical findings with extensive numerical experiments, using both least squares and hinge loss for our objective function, and demonstrate that our methods achieve state-of-the-art empirical support recovery, outperforming competing algorithms in settings with up to $p=10^4$.