Learning Visually Grounded Domain Ontologies via Embodied Conversation and Explanation

TL;DR

This paper presents a framework where an agent learns domain ontologies grounded in visual perception through embodied conversation and explanations, improving learning efficiency in low-resource scenarios.

Contribution

It introduces a novel learning approach combining explanations and corrective feedback to enhance ontology acquisition and visual recognition in low-resource settings.

Findings

Teacher-learner pairs with explanations learn more efficiently.

The framework improves the agent's understanding of domain ontologies.

Feedback-driven learning enhances visual recognition accuracy.

Abstract

In this paper, we offer a learning framework in which the agent's knowledge gaps are overcome through corrective feedback from a teacher whenever the agent explains its (incorrect) predictions. We test it in a low-resource visual processing scenario, in which the agent must learn to recognize distinct types of toy truck. The agent starts the learning process with no ontology about what types of trucks exist nor which parts they have, and a deficient model for recognizing those parts from visual input. The teacher's feedback to the agent's explanations addresses its lack of relevant knowledge in the ontology via a generic rule (e.g., "dump trucks have dumpers"), whereas an inaccurate part recognition is corrected by a deictic statement (e.g., "this is not a dumper"). The learner utilizes this feedback not only to improve its estimate of the hypothesis space of possible domain ontologies and probability distributions over them, but also to use those estimates to update its visual interpretation of the scene. Our experiments demonstrate that teacher-learner pairs utilizing explanations and corrections are more data-efficient than those without such a faculty.