Reagent: Converting Ordinary Webpages into Interactive Software Agents

TL;DR

Reagent is a novel system that transforms ordinary webpages into interactive agents capable of understanding natural language and pointing gestures to perform data visualization and analysis tasks without requiring webpage modifications.

Contribution

Reagent automatically extracts semantic information from arbitrary webpages and combines speech with pointing to enable natural interaction, overcoming limitations of prior specialized instrumentation.

Findings

Successfully applied to football statistics webpages

Enables natural language and gesture-based data manipulation

Proactively learns new terminology from users

Abstract

We introduce Reagent, a technology that readily converts ordinary webpages containing structured data into software agents with which one can interact naturally, via a combination of speech and pointing. Previous efforts to make webpage content manipulable by third-party software components in browsers or desktop applications have generally relied upon specialized instrumentation included in the webpages -- a practice that neither scales well nor applies to pre-existing webpages. In contrast, Reagent automatically captures semantic details and semantically-meaningful mouse events from arbitrary webpages that contain no pre-existing special instrumentation. Reagent combines these events with text transcriptions of user speech to derive and execute parameterized commands representing human intent. Thus, users may request various visualization or analytic operations to be performed on data displayed on a page by speaking to it and/or pointing to elements within it. When unable to infer translations between event labels and human terminology, Reagent proactively asks users for definitions and adds them to its dictionary. We demonstrate Reagent in the context of a collection of pre-existing webpages that contain football team and player statistics.