LEGO Co-builder: Exploring Fine-Grained Vision-Language Modeling for Multimodal LEGO Assembly Assistants

TL;DR

This paper introduces LEGO Co-builder, a benchmark for evaluating vision-language models on fine-grained LEGO assembly tasks, revealing current models' limitations in detailed spatial and state understanding.

Contribution

It presents a new hybrid benchmark dataset and evaluates leading models, highlighting gaps in fine-grained multimodal understanding for assembly tasks.

Findings

GPT-4o achieves a maximum F1 score of 40.54% in state detection.

Leading models struggle with fine-grained spatial reasoning.

The benchmark and tools are publicly released for future research.

Abstract

Vision-language models (VLMs) are facing the challenges of understanding and following multimodal assembly instructions, particularly when fine-grained spatial reasoning and precise object state detection are required. In this work, we explore LEGO Co-builder, a hybrid benchmark combining real-world LEGO assembly logic with programmatically generated multimodal scenes. The dataset captures stepwise visual states and procedural instructions, allowing controlled evaluation of instruction-following, object detection, and state detection. We introduce a unified framework and assess leading VLMs such as GPT-4o, Gemini, and Qwen-VL, under zero-shot and fine-tuned settings. Our results reveal that even advanced models like GPT-4o struggle with fine-grained assembly tasks, with a maximum F1 score of just 40.54\% on state detection, highlighting gaps in fine-grained visual understanding. We release the benchmark, codebase, and generation pipeline to support future research on multimodal assembly assistants grounded in real-world workflows.