On the Multi-turn Instruction Following for Conversational Web Agents

TL;DR

This paper introduces a new task called Conversational Web Navigation, supported by a novel dataset and a self-reflective memory-augmented planning framework, to improve multi-turn instruction following by LLM-powered web agents.

Contribution

The work presents a new dataset and a novel Self-MAP framework to enhance multi-turn instruction following in web agents, addressing context limitations and dependency issues.

Findings

Self-MAP outperforms baseline methods on MT-Mind2Web.

Memory and self-reflection improve task success rates.

Benchmark results demonstrate the effectiveness of the proposed approach.

Abstract

Web agents powered by Large Language Models (LLMs) have demonstrated remarkable abilities in planning and executing multi-step interactions within complex web-based environments, fulfilling a wide range of web navigation tasks. Despite these advancements, the potential for LLM-powered agents to effectively engage with sequential user instructions in real-world scenarios has not been fully explored. In this work, we introduce a new task of Conversational Web Navigation, which necessitates sophisticated interactions that span multiple turns with both the users and the environment, supported by a specially developed dataset named Multi-Turn Mind2Web (MT-Mind2Web). To tackle the limited context length of LLMs and the context-dependency issue of the conversational tasks, we further propose a novel framework, named self-reflective memory-augmented planning (Self-MAP), which employs memory utilization and self-reflection techniques. Extensive experiments are conducted to benchmark the MT-Mind2Web dataset, and validate the effectiveness of the proposed method.