Navigating WebAI: Training Agents to Complete Web Tasks with Large Language Models and Reinforcement Learning

TL;DR

This paper introduces a hybrid supervised and reinforcement learning approach for web navigation using large language models, improving understanding of HTML content and achieving better performance on the MiniWoB benchmark.

Contribution

It presents a novel combination of SL and RL techniques for web navigation, addressing HTML comprehension limitations and establishing new baseline results.

Findings

Outperforms previous SL methods with less data

Achieves 43.58% average accuracy with SL

Reaches 36.69% accuracy with multimodal RL

Abstract

Recent advancements in language models have demonstrated remarkable improvements in various natural language processing (NLP) tasks such as web navigation. Supervised learning (SL) approaches have achieved impressive performance while utilizing significantly less training data compared to previous methods. However, these SL-based models fall short when compared to reinforcement learning (RL) approaches, which have shown superior results. In this paper, we propose a novel approach that combines SL and RL techniques over the MiniWoB benchmark to leverage the strengths of both methods. We also address a critical limitation in previous models' understanding of HTML content, revealing a tendency to memorize target elements rather than comprehend the underlying structure. To rectify this, we propose methods to enhance true understanding and present a new baseline of results. Our experiments demonstrate that our approach outperforms previous SL methods on certain tasks using less data and narrows the performance gap with RL models, achieving 43.58\% average accuracy in SL and 36.69\% when combined with a multimodal RL approach. This study sets a new direction for future web navigation and offers insights into the limitations and potential of language modeling for computer tasks.