Explainable Behavior Cloning: Teaching Large Language Model Agents through Learning by Demonstration

TL;DR

This paper introduces EBC-LLMAgent, an explainable large language model-based agent that learns from demonstrations to autonomously interact with mobile apps, improving task success, generalization, and interpretability.

Contribution

It presents a novel framework combining behavior cloning with explainability in LLM agents for mobile app interaction, including new modules and a fusion technique.

Findings

High success rates in task completion across five apps

Effective generalization to unseen scenarios

Generation of meaningful explanations

Abstract

Autonomous mobile app interaction has become increasingly important with growing complexity of mobile applications. Developing intelligent agents that can effectively navigate and interact with mobile apps remains a significant challenge. In this paper, we propose an Explainable Behavior Cloning LLM Agent (EBC-LLMAgent), a novel approach that combines large language models (LLMs) with behavior cloning by learning demonstrations to create intelligent and explainable agents for autonomous mobile app interaction. EBC-LLMAgent consists of three core modules: Demonstration Encoding, Code Generation, and UI Mapping, which work synergistically to capture user demonstrations, generate executable codes, and establish accurate correspondence between code and UI elements. We introduce the Behavior Cloning Chain Fusion technique to enhance the generalization capabilities of the agent. Extensive experiments on five popular mobile applications from diverse domains demonstrate the superior performance of EBC-LLMAgent, achieving high success rates in task completion, efficient generalization to unseen scenarios, and the generation of meaningful explanations.