VLMgineer: Vision Language Models as Robotic Toolsmiths

TL;DR

VLMgineer leverages vision language models and evolutionary search to automatically co-design tools and action plans, enabling robots to solve manipulation tasks more effectively and creatively.

Contribution

This work introduces VLMgineer, a novel framework combining VLMs and evolutionary search for automated tool design and use in robotics.

Findings

VLMgineer outperforms human-designed tools in manipulation tasks.

The framework discovers innovative tools and policies for diverse scenarios.

It transforms complex robotics problems into simple, effective executions.

Abstract

Tool design and use reflect the ability to understand and manipulate the physical world through creativity, planning, and foresight. As such, these capabilities are often regarded as measurable indicators of intelligence across biological species. While much of today's research on robotic intelligence focuses on generating better controllers, inventing smarter tools offers a complementary form of physical intelligence: shifting the onus of problem-solving onto the tool's design. Given the vast and impressive common-sense, reasoning, and creative capabilities of today's foundation models, we investigate whether these models can provide useful priors to automatically design and effectively wield such tools? We present VLMgineer, a framework that harnesses the code generation abilities of vision language models (VLMs) together with evolutionary search to iteratively co-design physical tools and the action plans that operate them to perform a task. We evaluate VLMgineer on a diverse new benchmark of everyday manipulation scenarios that demand creative tool design and use. Across this suite, VLMgineer consistently discovers tools and policies that solve tasks more effectively and innovatively, transforming challenging robotics problems into straightforward executions. It also outperforms VLM-generated designs from human specifications and existing human-crafted tools for everyday tasks. To facilitate future research on automated tool invention, we will release our benchmark and code.