Learning Visual-Audio Representations for Voice-Controlled Robots

TL;DR

This paper introduces a novel visual-audio representation learning pipeline for voice-controlled robots that improves task performance with fewer labels and better generalization across different robots and tasks.

Contribution

The authors propose a new pipeline that learns visual-audio representations and uses reinforcement learning, reducing label dependence and enhancing adaptability across platforms.

Findings

Outperforms previous methods with fewer labels

Works across various robots and tasks

Self-improves in unseen scenarios with limited new data

Abstract

Based on the recent advancements in representation learning, we propose a novel pipeline for task-oriented voice-controlled robots with raw sensor inputs. Previous methods rely on a large number of labels and task-specific reward functions. Not only can such an approach hardly be improved after the deployment, but also has limited generalization across robotic platforms and tasks. To address these problems, our pipeline first learns a visual-audio representation (VAR) that associates images and sound commands. Then the robot learns to fulfill the sound command via reinforcement learning using the reward generated by the VAR. We demonstrate our approach with various sound types, robots, and tasks. We show that our method outperforms previous work with much fewer labels. We show in both the simulated and real-world experiments that the system can self-improve in previously unseen scenarios given a reasonable number of newly labeled data.