Theoretical Analysis of Measure Consistency Regularization for Partially Observed Data

TL;DR

This paper provides a theoretical understanding of Measure Consistency Regularization (MCR) in partially observed data scenarios, explaining its benefits, limitations, and proposing an early stopping method to optimize its effectiveness.

Contribution

It offers the first theoretical analysis of MCR, identifying the key factors behind its generalization benefits and proposing a novel training protocol with empirical validation.

Findings

MCR improves imputation quality under certain conditions.

The generalization advantage of MCR is not guaranteed in imperfect training regimes.

A duality gap-based early stopping method enhances MCR performance.

Abstract

The problem of corrupted data, missing features, or missing modalities continues to plague the modern machine learning landscape. To address this issue, a class of regularization methods that enforce consistency between imputed and fully observed data has emerged as a promising approach for improving model generalization, particularly in partially observed settings. We refer to this class of methods as Measure Consistency Regularization (MCR). Despite its empirical success in various applications, such as image inpainting, data imputation and semi-supervised learning, a fundamental understanding of the theoretical underpinnings of MCR remains limited. This paper bridges this gap by offering theoretical insights into why, when, and how MCR enhances imputation quality under partial observability, viewed through the lens of neural network distance. Our theoretical analysis identifies the term responsible for MCR's generalization advantage and extends to the imperfect training regime, demonstrating that this advantage is not always guaranteed. Guided by these insights, we propose a novel training protocol that monitors the duality gap to determine an early stopping point that preserves the generalization benefit. We then provide detailed empirical evidence to support our theoretical claims and to show the effectiveness and accuracy of our proposed stopping condition. We further provide a set of real-world data simulations to show the versatility of MCR under different model architectures designed for different data sources.