VSGM -- Enhance robot task understanding ability through visual semantic graph

TL;DR

This paper introduces VSGM, a novel visual semantic graph memory approach that enhances robot understanding of visual and language cues, leading to improved task success rates in indoor household tasks.

Contribution

The paper proposes VSGM, a new method using semantic graphs and graph neural networks to improve robot visual and language understanding for task execution.

Findings

Task success rate improved by 6-10% with VSGM.

Enhanced visual understanding through semantic graph representation.

Effective mapping of objects to egocentric maps.

Abstract

In recent years, developing AI for robotics has raised much attention. The interaction of vision and language of robots is particularly difficult. We consider that giving robots an understanding of visual semantics and language semantics will improve inference ability. In this paper, we propose a novel method-VSGM (Visual Semantic Graph Memory), which uses the semantic graph to obtain better visual image features, improve the robot's visual understanding ability. By providing prior knowledge of the robot and detecting the objects in the image, it predicts the correlation between the attributes of the object and the objects and converts them into a graph-based representation; and mapping the object in the image to be a top-down egocentric map. Finally, the important object features of the current task are extracted by Graph Neural Networks. The method proposed in this paper is verified in the ALFRED (Action Learning From Realistic Environments and Directives) dataset. In this dataset, the robot needs to perform daily indoor household tasks following the required language instructions. After the model is added to the VSGM, the task success rate can be improved by 6~10%.