Manual2Skill: Learning to Read Manuals and Acquire Robotic Skills for Furniture Assembly Using Vision-Language Models

TL;DR

Manual2Skill enables robots to interpret high-level manual instructions and perform complex furniture assembly tasks by integrating vision-language models, hierarchical graph construction, pose estimation, and motion planning, demonstrating practical real-world applications.

Contribution

The paper introduces Manual2Skill, a novel framework that combines vision-language models, hierarchical assembly graphs, pose estimation, and motion planning for robotic furniture assembly from manuals.

Findings

Successfully assembled IKEA furniture items in real-world tests.

Enhanced robot understanding and execution of complex, long-horizon tasks.

Demonstrated practical application of vision-language models in robotics.

Abstract

Humans possess an extraordinary ability to understand and execute complex manipulation tasks by interpreting abstract instruction manuals. For robots, however, this capability remains a substantial challenge, as they cannot interpret abstract instructions and translate them into executable actions. In this paper, we present Manual2Skill, a novel framework that enables robots to perform complex assembly tasks guided by high-level manual instructions. Our approach leverages a Vision-Language Model (VLM) to extract structured information from instructional images and then uses this information to construct hierarchical assembly graphs. These graphs represent parts, subassemblies, and the relationships between them. To facilitate task execution, a pose estimation model predicts the relative 6D poses of components at each assembly step. At the same time, a motion planning module generates actionable sequences for real-world robotic implementation. We demonstrate the effectiveness of Manual2Skill by successfully assembling several real-world IKEA furniture items. This application highlights its ability to manage long-horizon manipulation tasks with both efficiency and precision, significantly enhancing the practicality of robot learning from instruction manuals. This work marks a step forward in advancing robotic systems capable of understanding and executing complex manipulation tasks in a manner akin to human capabilities.Project Page: https://owensun2004.github.io/Furniture-Assembly-Web/