"How to best say it?" : Translating Directives in Machine Language into Natural Language in the Blocks World

TL;DR

This paper presents a method to translate machine-generated block placement commands into natural language expressions that are easier for humans to understand, using an algorithm that generates and evaluates multiple alternatives based on comprehension ease.

Contribution

It introduces a novel algorithm that transforms machine directives into natural language in ECI-space and employs a cost function derived from user studies to select the most comprehensible expression.

Findings

Generated expressions improve human understanding of machine directives.

The cost function effectively predicts utterance-to-action timings.

Algorithm produces multiple alternative expressions for optimal clarity.

Abstract

We propose a method to generate optimal natural language for block placement directives generated by a machine's planner during human-agent interactions in the blocks world. A non user-friendly machine directive, e.g., move(ObjId, toPos), is transformed into visually and contextually grounded referring expressions that are much easier for the user to comprehend. We describe an algorithm that progressively and generatively transforms the machine's directive in ECI (Elementary Composable Ideas)-space, generating many alternative versions of the directive. We then define a cost function to evaluate the ease of comprehension of these alternatives and select the best option. The parameters for this cost function were derived empirically from a user study that measured utterance-to-action timings.