Exploring Plan Space through Conversation: An Agentic Framework for LLM-Mediated Explanations in Planning

TL;DR

This paper introduces a multi-agent LLM framework for interactive, context-dependent explanations in planning, demonstrated through goal-conflict explanations and user studies to enhance understanding and trust.

Contribution

It presents a novel agentic LLM architecture for flexible, conversational explanations in planning, adaptable to various explanation frameworks and user needs.

Findings

LLM-based interaction improves user understanding over template-based methods.

The framework supports goal-conflict explanations effectively.

User trust increases with interactive, context-aware explanations.

Abstract

When automating plan generation for a real-world sequential decision problem, the goal is often not to replace the human planner, but to facilitate an iterative reasoning and elicitation process, where the human's role is to guide the AI planner according to their preferences and expertise. In this context, explanations that respond to users' questions are crucial to improve their understanding of potential solutions and increase their trust in the system. To enable natural interaction with such a system, we present a multi-agent Large Language Model (LLM) architecture that is agnostic to the explanation framework and enables user- and context-dependent interactive explanations. We also describe an instantiation of this framework for goal-conflict explanations, which we use to conduct a user study comparing the LLM-powered interaction with a baseline template-based explanation interface.