TILT: Target-induced loss tilting under covariate shift

TL;DR

TILT is a novel method for unsupervised domain adaptation under covariate shift that improves target-domain performance by decomposing the predictor and penalizing auxiliary components, with theoretical guarantees and empirical success.

Contribution

Introduces TILT, a new objective for domain adaptation that implicitly performs importance weighting and provides theoretical and empirical advantages.

Findings

TILT outperforms source-only, importance weighting, and density-ratio baselines.

Provides finite-sample oracle inequality and end-to-end guarantees.

Demonstrates effectiveness on regression and CIFAR-100 distillation tasks.

Abstract

We introduce and analyze Target-Induced Loss Tilting (TILT) for unsupervised domain adaptation under covariate shift. It is based on a novel objective function that decomposes the source predictor as $f+b$, fits $f+b$ on labeled source data while simultaneously penalizing the auxiliary component $b$ on unlabeled target inputs. The resulting fit $f$ is deployed as the final target predictor. At the population level, we show that this target-side penalty implicitly induces relative importance weighting at the population level, but in terms of an estimand $b^*_f$ that is self-localized to the current error, and remains uniformly bounded for any source-target pair (even those with disjoint supports). We prove a general finite-sample oracle inequality on the excess risk, and use it to give an end-to-end guarantee for training with sparse ReLU networks. Experiments on controlled regression problems and shifted CIFAR-100 distillation show that TILT improves target-domain performance over source-only training, exact importance weighting, and relative density-ratio baselines, with a stable dependence on the regularization parameter.