Leolani: a reference machine with a theory of mind for social communication

TL;DR

This paper introduces Leolani, a social robot that models a theory of mind to manage conflicting information, uncertainties, and social interactions by learning from perceptions and conversations with humans.

Contribution

It presents a novel robot architecture that integrates perception, speech, and provenance tracking to simulate a theory of mind in social communication.

Findings

The robot can resolve conflicting information through social interactions.

Leolani demonstrates awareness of its perceptions and sources.

The system enables more natural and context-aware human-robot communication.

Abstract

Our state of mind is based on experiences and what other people tell us. This may result in conflicting information, uncertainty, and alternative facts. We present a robot that models relativity of knowledge and perception within social interaction following principles of the theory of mind. We utilized vision and speech capabilities on a Pepper robot to build an interaction model that stores the interpretations of perceptions and conversations in combination with provenance on its sources. The robot learns directly from what people tell it, possibly in relation to its perception. We demonstrate how the robot's communication is driven by hunger to acquire more knowledge from and on people and objects, to resolve uncertainties and conflicts, and to share awareness of the per- ceived environment. Likewise, the robot can make reference to the world and its knowledge about the world and the encounters with people that yielded this knowledge.