Training a Multilingual Sportscaster: Using Perceptual Context to Learn Language

TL;DR

This paper introduces a framework that learns to interpret and generate multilingual sportscasting language from perceptual context alone, without language-specific prior knowledge, demonstrated on simulated robot soccer games in English and Korean.

Contribution

It presents a novel perceptual supervision approach for multilingual language learning, including a new algorithm for identifying worth-describing events, and shows promising results in a simulated domain.

Findings

System learns to associate comments with events without language prior

Generated commentaries are of reasonable quality, comparable to humans in some cases

Supports both parsing and generation in multiple languages

Abstract

We present a novel framework for learning to interpret and generate language using only perceptual context as supervision. We demonstrate its capabilities by developing a system that learns to sportscast simulated robot soccer games in both English and Korean without any language-specific prior knowledge. Training employs only ambiguous supervision consisting of a stream of descriptive textual comments and a sequence of events extracted from the simulation trace. The system simultaneously establishes correspondences between individual comments and the events that they describe while building a translation model that supports both parsing and generation. We also present a novel algorithm for learning which events are worth describing. Human evaluations of the generated commentaries indicate they are of reasonable quality and in some cases even on par with those produced by humans for our limited domain.