Language-Induced Priors for Domain Adaptation

TL;DR

This paper introduces a novel probabilistic framework that uses textual descriptions of target domains to improve source relevance identification in domain adaptation, especially in data-scarce scenarios.

Contribution

It proposes the Language-Induced Prior (LIP), leveraging large language models to guide source selection and improve adaptation performance.

Findings

LIP guides source relevance in weak signal regimes.

The estimator matches an oracle MSE under correct prior.

Framework is validated on Gaussian, predictive, and prescriptive tasks.

Abstract

Domain adaptation faces a fundamental paradox in the cold-start regime. When target data is scarce, statistical methods fail to distinguish relevant source domains from irrelevant ones, which often leads to negative transfer. In this paper, we address this challenge by leveraging expert textual descriptions of the target domain, a resource that is often available but overlooked. We propose a probabilistic framework that translates these semantic descriptions into a choice model, namely a Language-Induced Prior (LIP), that learns the preferences from a pretrained Large Language Model (LLM). The LIP is then integrated into an Expectation-Maximization algorithm to identify source relevance. Methodologically, this framework is compatible with any parametric model where a likelihood is available. It allows the LIP to guide the selection of sources when target signals are weak, while gradually refining these choices as samples accumulate. Theoretically, we prove that the estimator roughly matches an oracle cold-start MSE under a correct prior, while remaining asymptotically consistent regardless of the quality of the LIP. Empirically, we validated the framework on a descriptive (Gaussian estimation), a predictive (C-MAPSS dataset), and a prescriptive task (MuJoCo hopper).