CheckManual: A New Challenge and Benchmark for Manual-based Appliance Manipulation

TL;DR

CheckManual introduces a new benchmark for robot manipulation of appliances based on understanding detailed manuals, addressing the gap in manual comprehension for complex tasks.

Contribution

It presents the first manual-based appliance manipulation benchmark and a planning model, enabling evaluation of manual comprehension in robotic manipulation.

Findings

Developed a large model-assisted manual data generation pipeline.

Established novel challenges and metrics for manual-based manipulation.

Created a simulator environment for benchmarking.

Abstract

Correct use of electrical appliances has significantly improved human life quality. Unlike simple tools that can be manipulated with common sense, different parts of electrical appliances have specific functions defined by manufacturers. If we want the robot to heat bread by microwave, we should enable them to review the microwave manual first. From the manual, it can learn about component functions, interaction methods, and representative task steps about appliances. However, previous manual-related works remain limited to question-answering tasks while existing manipulation researchers ignore the manual's important role and fail to comprehend multi-page manuals. In this paper, we propose the first manual-based appliance manipulation benchmark CheckManual. Specifically, we design a large model-assisted human-revised data generation pipeline to create manuals based on CAD appliance models. With these manuals, we establish novel manual-based manipulation challenges, metrics, and simulator environments for model performance evaluation. Furthermore, we propose the first manual-based manipulation planning model ManualPlan to set up a group of baselines for the CheckManual benchmark.