Learning Personalized Agents from Human Feedback

TL;DR

This paper introduces PAHF, a framework enabling AI agents to learn and adapt to individual user preferences over time through online interaction and explicit memory, improving personalization and responsiveness.

Contribution

The paper proposes PAHF, a novel continual learning framework that uses explicit memory and dual feedback channels for personalized agents to adapt to evolving user preferences.

Findings

PAHF learns faster than baseline methods.

PAHF outperforms no-memory and single-channel baselines.

PAHF effectively adapts to preference shifts.

Abstract

Modern AI agents are powerful but often fail to align with the idiosyncratic, evolving preferences of individual users. Prior approaches typically rely on static datasets, either training implicit preference models on interaction history or encoding user profiles in external memory. However, these approaches struggle with new users and with preferences that change over time. We introduce Personalized Agents from Human Feedback (PAHF), a framework for continual personalization in which agents learn online from live interaction using explicit per-user memory. PAHF operationalizes a three-step loop: (1) seeking pre-action clarification to resolve ambiguity, (2) grounding actions in preferences retrieved from memory, and (3) integrating post-action feedback to update memory when preferences drift. To evaluate this capability, we develop a four-phase protocol and two benchmarks in embodied manipulation and online shopping. These benchmarks quantify an agent's ability to learn initial preferences from scratch and subsequently adapt to persona shifts. Our theoretical analysis and empirical results show that integrating explicit memory with dual feedback channels is critical: PAHF learns substantially faster and consistently outperforms both no-memory and single-channel baselines, reducing initial personalization error and enabling rapid adaptation to preference shifts.