Towards an Indexical Model of Situated Language Comprehension for Cognitive Agents in Physical Worlds

TL;DR

This paper introduces an indexical computational model for situated language comprehension in cognitive agents, integrating perceptions, knowledge, and experiences to resolve ambiguities in real-world interactions.

Contribution

It presents a novel indexical model translating linguistic symbols into modal representations, enhancing understanding in agents through multiple information sources.

Findings

Model helps resolve ambiguities in referring expressions

Incorporates perceptions and knowledge for better comprehension

Supports linguistic interactions in a learning agent

Abstract

We propose a computational model of situated language comprehension based on the Indexical Hypothesis that generates meaning representations by translating amodal linguistic symbols to modal representations of beliefs, knowledge, and experience external to the linguistic system. This Indexical Model incorporates multiple information sources, including perceptions, domain knowledge, and short-term and long-term experiences during comprehension. We show that exploiting diverse information sources can alleviate ambiguities that arise from contextual use of underspecific referring expressions and unexpressed argument alternations of verbs. The model is being used to support linguistic interactions in Rosie, an agent implemented in Soar that learns from instruction.