droidlet: modular, heterogenous, multi-modal agents

TL;DR

Droidlet is an open-source, modular platform that integrates perception, language, and action, enabling agents to learn from real-world interactions by combining large-scale datasets with robotics heuristics.

Contribution

It introduces a novel, flexible agent architecture that unifies perception, language, and action, facilitating multi-modal learning from static datasets and real-world interactions.

Findings

Integrated perception, language, and action in a single platform

Supports learning from static datasets and real-world interactions

Open-source tool for developing multi-modal agents

Abstract

In recent years, there have been significant advances in building end-to-end Machine Learning (ML) systems that learn at scale. But most of these systems are: (a) isolated (perception, speech, or language only); (b) trained on static datasets. On the other hand, in the field of robotics, large-scale learning has always been difficult. Supervision is hard to gather and real world physical interactions are expensive. In this work we introduce and open-source droidlet, a modular, heterogeneous agent architecture and platform. It allows us to exploit both large-scale static datasets in perception and language and sophisticated heuristics often used in robotics; and provides tools for interactive annotation. Furthermore, it brings together perception, language and action onto one platform, providing a path towards agents that learn from the richness of real world interactions.