Mapping Natural Language Commands to Web Elements

TL;DR

This paper introduces a new task of mapping natural language commands to web elements, supported by a large dataset and baseline models, to improve language grounding in web environments.

Contribution

It presents the first dataset of over 50,000 commands for grounding language in web environments and analyzes baseline models for this task.

Findings

Dataset of 50,000+ commands collected

Baseline models analyzed for language grounding

Captures functional, relational, and visual reasoning phenomena

Abstract

The web provides a rich, open-domain environment with textual, structural, and spatial properties. We propose a new task for grounding language in this environment: given a natural language command (e.g., "click on the second article"), choose the correct element on the web page (e.g., a hyperlink or text box). We collected a dataset of over 50,000 commands that capture various phenomena such as functional references (e.g. "find who made this site"), relational reasoning (e.g. "article by john"), and visual reasoning (e.g. "top-most article"). We also implemented and analyzed three baseline models that capture different phenomena present in the dataset.