Multimodal Semantic Simulations of Linguistically Underspecified Motion Events

TL;DR

This paper presents a system that generates 3D visual simulations of natural language motion descriptions using semantic models to ensure minimal constraints are satisfied, integrating physical knowledge into the process.

Contribution

It introduces a novel system that combines formal semantic models with 3D simulation to accurately depict linguistically underspecified motion events.

Findings

Effective generation of 3D simulations from natural language

Semantic models ensure minimal constraints are met

Integration of physical knowledge improves realism

Abstract

In this paper, we describe a system for generating three-dimensional visual simulations of natural language motion expressions. We use a rich formal model of events and their participants to generate simulations that satisfy the minimal constraints entailed by the associated utterance, relying on semantic knowledge of physical objects and motion events. This paper outlines technical considerations and discusses implementing the aforementioned semantic models into such a system.