What Should the System Do Next?: Operative Action Captioning for Estimating System Actions

TL;DR

This paper introduces a new task called operative action captioning for robots, which involves generating verbal descriptions of actions to change current states to target states, aiding human-assisting systems.

Contribution

The study proposes a novel captioning task for estimating system actions, constructs a dataset with paired images and action captions, and demonstrates improved action estimation using scene-graph auxiliary tasks.

Findings

The system successfully describes operative actions between states.

Scene-graph prediction improves caption quality.

The dataset enables training for state-change action captioning.

Abstract

Such human-assisting systems as robots need to correctly understand the surrounding situation based on observations and output the required support actions for humans. Language is one of the important channels to communicate with humans, and the robots are required to have the ability to express their understanding and action planning results. In this study, we propose a new task of operative action captioning that estimates and verbalizes the actions to be taken by the system in a human-assisting domain. We constructed a system that outputs a verbal description of a possible operative action that changes the current state to the given target state. We collected a dataset consisting of two images as observations, which express the current state and the state changed by actions, and a caption that describes the actions that change the current state to the target state, by crowdsourcing in daily life situations. Then we constructed a system that estimates operative action by a caption. Since the operative action's caption is expected to contain some state-changing actions, we use scene-graph prediction as an auxiliary task because the events written in the scene graphs correspond to the state changes. Experimental results showed that our system successfully described the operative actions that should be conducted between the current and target states. The auxiliary tasks that predict the scene graphs improved the quality of the estimation results.