A Human-Centered Data-Driven Planner-Actor-Critic Architecture via Logic Programming

TL;DR

This paper introduces PACMAN, a unified human-centered planning and learning framework combining symbolic knowledge, reinforcement learning, and human feedback to enable faster, more robust policy development in agents.

Contribution

It presents the first integrated architecture that combines knowledge-based planning, RL, and human teaching for improved policy learning.

Findings

PACMAN achieves rapid early learning and convergence.

It is robust to inconsistent and infrequent human feedback.

Demonstrates significant improvements over traditional RL approaches.

Abstract

Recent successes of Reinforcement Learning (RL) allow an agent to learn policies that surpass human experts but suffers from being time-hungry and data-hungry. By contrast, human learning is significantly faster because prior and general knowledge and multiple information resources are utilized. In this paper, we propose a Planner-Actor-Critic architecture for huMAN-centered planning and learning (PACMAN), where an agent uses its prior, high-level, deterministic symbolic knowledge to plan for goal-directed actions, and also integrates the Actor-Critic algorithm of RL to fine-tune its behavior towards both environmental rewards and human feedback. This work is the first unified framework where knowledge-based planning, RL, and human teaching jointly contribute to the policy learning of an agent. Our experiments demonstrate that PACMAN leads to a significant jump-start at the early stage of learning, converges rapidly and with small variance, and is robust to inconsistent, infrequent, and misleading feedback.