Anytime and Difficulty-Adaptive PAC-Bayes for Constrained Density-Ratio Network with Continual Learning Guarantees

TL;DR

This paper introduces a unified framework using a constrained density-ratio network with PAC-Bayes guarantees for learning under covariate shift, supporting importance weighting and providing anytime generalization certificates.

Contribution

It develops a novel density-ratio network with integral constraints and instantiates PAC-Bayes bounds in fixed-time and anytime regimes for covariate shift adaptation.

Findings

Reduces target 0/1 loss compared to baseline methods.

Produces a calibrated covariate ratio on real data.

Attains an anytime generalization certificate with empirical validation.

Abstract

A unified framework for learning under covariate shift is presented, in which a constrained density-ratio network approximates the Radon-Nikodym derivative $r^\star = dP/dQ$ from source $Q$ to target $P$, supports an importance-weighted empirical risk, and feeds an anytime PAC-Bayes generalization certificate. A change-of-measure identity decomposes the gap between target risk and importance-weighted source risk into a ratio-bias term, controlled by the $L^2(Q)$ closeness of the learned ratio to $r^\star$, and a generalization-gap term, controlled by the variability of the weighted loss. Three structural identities of a Radon-Nikodym derivative, normalization, moment matching, and a second-moment penalty controlling the effective sample size, are imposed as hard integral constraints through an augmented-Lagrangian scheme. PAC-Bayes is then instantiated on the weighted risk in a fixed-time regime that yields Bernoulli-KL bounds, a KL-regularized objective whose minimizer is the network-weighted Gibbs posterior, and a stability statement on $L^2(Q)$ perturbations of the learned ratio, and in an anytime regime that builds a time-uniform certificate by geometric peeling across epochs. A pre-registered two-campaign protocol combining a patch test against analytic ground truth with a real-data deployment under intrinsic distribution shift validates the framework. The network produces a calibrated covariate ratio on real data, reduces the target $0/1$ loss relative to unweighted empirical risk minimization and to classical direct ratio-estimation baselines, and attains the anytime certificate as the construction promises. A single pre-registered failure of the fixed-time coverage claim is recorded, with per-split coverage aligning one-to-one with the magnitude of the label shift, confirming that the covariate-only assumption is operationally tight rather than a defect of the certificate.