EL3DD: Extended Latent 3D Diffusion for Language Conditioned Multitask Manipulation

TL;DR

This paper introduces EL3DD, a diffusion-based visuomotor policy that integrates visual and language inputs to improve robotic manipulation in human environments, demonstrating enhanced multitask performance.

Contribution

It extends existing diffusion models with better embeddings and techniques for language-conditioned robotic manipulation, advancing multitask capabilities.

Findings

Improved success rates on the CALVIN dataset for manipulation tasks.

Enhanced performance in executing multiple tasks sequentially.

Reinforced the effectiveness of diffusion models in robotic control.

Abstract

Acting in human environments is a crucial capability for general-purpose robots, necessitating a robust understanding of natural language and its application to physical tasks. This paper seeks to harness the capabilities of diffusion models within a visuomotor policy framework that merges visual and textual inputs to generate precise robotic trajectories. By employing reference demonstrations during training, the model learns to execute manipulation tasks specified through textual commands within the robot's immediate environment. The proposed research aims to extend an existing model by leveraging improved embeddings, and adapting techniques from diffusion models for image generation. We evaluate our methods on the CALVIN dataset, proving enhanced performance on various manipulation tasks and an increased long-horizon success rate when multiple tasks are executed in sequence. Our approach reinforces the usefulness of diffusion models and contributes towards general multitask manipulation.