Learning Human-Like RL Agents Through Trajectory Optimization With Action Quantization

TL;DR

This paper introduces Macro Action Quantization (MAQ), a novel framework that enhances the human-likeness of reinforcement learning agents by aligning their trajectories with human behavior through trajectory optimization and macro actions.

Contribution

The paper proposes MAQ, a new method that distills human demonstrations into macro actions to produce more human-like RL agents, improving interpretability and trustworthiness.

Findings

MAQ significantly increases trajectory similarity scores.

MAQ achieves the highest human-likeness rankings in evaluations.

MAQ can be integrated into various RL algorithms.

Abstract

Human-like agents have long been one of the goals in pursuing artificial intelligence. Although reinforcement learning (RL) has achieved superhuman performance in many domains, relatively little attention has been focused on designing human-like RL agents. As a result, many reward-driven RL agents often exhibit unnatural behaviors compared to humans, raising concerns for both interpretability and trustworthiness. To achieve human-like behavior in RL, this paper first formulates human-likeness as trajectory optimization, where the objective is to find an action sequence that closely aligns with human behavior while also maximizing rewards, and adapts the classic receding-horizon control to human-like learning as a tractable and efficient implementation. To achieve this, we introduce Macro Action Quantization (MAQ), a human-like RL framework that distills human demonstrations into macro actions via Vector-Quantized VAE. Experiments on D4RL Adroit benchmarks show that MAQ significantly improves human-likeness, increasing trajectory similarity scores, and achieving the highest human-likeness rankings among all RL agents in the human evaluation study. Our results also demonstrate that MAQ can be easily integrated into various off-the-shelf RL algorithms, opening a promising direction for learning human-like RL agents. Our code is available at https://rlg.iis.sinica.edu.tw/papers/MAQ.