Distribution-Free Matrix Prediction Under Arbitrary Missing Pattern

TL;DR

This paper addresses the challenge of conformalized entry prediction in matrices with arbitrary missing patterns, proposing efficient algorithms that ensure coverage validity and analyzing the impact of missingness on accuracy.

Contribution

It introduces two novel algorithms for matrix prediction with missing data, providing theoretical guarantees and empirical validation of their effectiveness.

Findings

Algorithms effectively safeguard coverage in missing data scenarios

Missingness impacts prediction accuracy quantitatively

Proven fundamental limits for matrix prediction under missing patterns

Abstract

This paper studies the open problem of conformalized entry prediction in a row/column-exchangeable matrix. The matrix setting presents novel and unique challenges, but there exists little work on this interesting topic. We meticulously define the problem, differentiate it from closely related problems, and rigorously delineate the boundary between achievable and impossible goals. We then propose two practical algorithms. The first method provides a fast emulation of the full conformal prediction, while the second method leverages the technique of algorithmic stability for acceleration. Both methods are computationally efficient and can effectively safeguard coverage validity in presence of arbitrary missing pattern. Further, we quantify the impact of missingness on prediction accuracy and establish fundamental limit results. Empirical evidence from synthetic and real-world data sets corroborates the superior performance of our proposed methods.