Acquiring Grounded Representations of Words with Situated Interactive Instruction

TL;DR

This paper introduces a method for robots to learn grounded word representations through interactive, situated instructions from humans, enabling efficient acquisition of perceptual, semantic, and procedural knowledge.

Contribution

The work presents a novel interactive learning approach for grounded language acquisition implemented in Soar and tested on a robotic arm in a tabletop environment.

Findings

Effective learning of diverse knowledge types

Robust grounded word representations acquired

Improved efficiency through interactive instruction

Abstract

We present an approach for acquiring grounded representations of words from mixed-initiative, situated interactions with a human instructor. The work focuses on the acquisition of diverse types of knowledge including perceptual, semantic, and procedural knowledge along with learning grounded meanings. Interactive learning allows the agent to control its learning by requesting instructions about unknown concepts, making learning efficient. Our approach has been instantiated in Soar and has been evaluated on a table-top robotic arm capable of manipulating small objects.