Hierarchical Reward Design from Language: Enhancing Alignment of Agent Behavior with Human Specifications

TL;DR

This paper introduces a hierarchical reward design framework from language to improve AI alignment with human preferences, enabling agents to better understand and follow complex, nuanced specifications in long-horizon tasks.

Contribution

It proposes HRDL for hierarchical reward specification and L2HR for translating language into hierarchical rewards, advancing human-aligned reinforcement learning.

Findings

Agents trained with L2HR better adhere to human specifications.

L2HR enables effective task completion with nuanced behavioral alignment.

The approach improves hierarchical RL's ability to interpret complex language instructions.

Abstract

When training artificial intelligence (AI) to perform tasks, humans often care not only about whether a task is completed but also how it is performed. As AI agents tackle increasingly complex tasks, aligning their behavior with human-provided specifications becomes critical for responsible AI deployment. Reward design provides a direct channel for such alignment by translating human expectations into reward functions that guide reinforcement learning (RL). However, existing methods are often too limited to capture nuanced human preferences that arise in long-horizon tasks. Hence, we introduce Hierarchical Reward Design from Language (HRDL): a problem formulation that extends classical reward design to encode richer behavioral specifications for hierarchical RL agents. We further propose Language to Hierarchical Rewards (L2HR) as a solution to HRDL. Experiments show that AI agents trained with rewards designed via L2HR not only complete tasks effectively but also better adhere to human specifications. Together, HRDL and L2HR advance the research on human-aligned AI agents.