Learning What Matters Now: Dynamic Preference Inference under Contextual Shifts

TL;DR

This paper introduces Dynamic Preference Inference (DPI), a framework enabling agents to adapt to changing preferences in decision-making tasks by inferring latent preferences and conditioning policies accordingly, outperforming static approaches.

Contribution

The paper proposes DPI, a novel probabilistic framework that infers and adapts to shifting preferences in sequential decision-making, integrating it with a preference-conditioned actor-critic.

Findings

DPI outperforms fixed-weight baselines in environments with changing objectives.

DPI effectively infers latent preferences during regime shifts.

DPI improves post-shift performance across various environments.

Abstract

Humans often juggle multiple, sometimes conflicting objectives and shift their priorities as circumstances change, rather than following a fixed objective function. In contrast, most computational decision-making and multi-objective RL methods assume static preference weights or a known scalar reward. In this work, we study sequential decision-making problem when these preference weights are unobserved latent variables that drift with context. Specifically, we propose Dynamic Preference Inference (DPI), a cognitively inspired framework in which an agent maintains a probabilistic belief over preference weights, updates this belief from recent interaction, and conditions its policy on inferred preferences. We instantiate DPI as a variational preference inference module trained jointly with a preference-conditioned actor-critic, using vector-valued returns as evidence about latent trade-offs. In queueing, maze, and multi-objective continuous-control environments with event-driven changes in objectives, DPI adapts its inferred preferences to new regimes and achieves higher post-shift performance than fixed-weight and heuristic envelope baselines.