WorldAPIs: The World Is Worth How Many APIs? A Thought Experiment

TL;DR

This paper explores the diversity of APIs needed for embodied AI agents to perform a wide range of tasks, proposing a framework to induce and analyze these APIs based on wikiHow instructions, highlighting the gap in current simulators.

Contribution

It introduces a novel framework for inducing and analyzing APIs for embodied agents using large language models and wikiHow data, emphasizing the need for richer simulated environments.

Findings

Induced over 300 APIs from a small subset of wikiHow tutorials.

Reused existing APIs and fabricated new ones to cover diverse tasks.

Current simulators support only a small fraction of these APIs.

Abstract

AI systems make decisions in physical environments through primitive actions or affordances that are accessed via API calls. While deploying AI agents in the real world involves numerous high-level actions, existing embodied simulators offer a limited set of domain-salient APIs. This naturally brings up the questions: how many primitive actions (APIs) are needed for a versatile embodied agent, and what should they look like? We explore this via a thought experiment: assuming that wikiHow tutorials cover a wide variety of human-written tasks, what is the space of APIs needed to cover these instructions? We propose a framework to iteratively induce new APIs by grounding wikiHow instruction to situated agent policies. Inspired by recent successes in large language models (LLMs) for embodied planning, we propose a few-shot prompting to steer GPT-4 to generate Pythonic programs as agent policies and bootstrap a universe of APIs by 1) reusing a seed set of APIs; and then 2) fabricate new API calls when necessary. The focus of this thought experiment is on defining these APIs rather than their executability. We apply the proposed pipeline on instructions from wikiHow tutorials. On a small fraction (0.5%) of tutorials, we induce an action space of 300+ APIs necessary for capturing the rich variety of tasks in the physical world. A detailed automatic and human analysis of the induction output reveals that the proposed pipeline enables effective reuse and creation of APIs. Moreover, a manual review revealed that existing simulators support only a small subset of the induced APIs (9 of the top 50 frequent APIs), motivating the development of action-rich embodied environments.