QueryCAD: Grounded Question Answering for CAD Models

TL;DR

QueryCAD is a novel system that enables natural language question answering on CAD models, facilitating information extraction and integration into robotic automation and deep learning frameworks.

Contribution

It introduces the first CAD question answering system, including a new open-vocabulary segmentation model and a benchmark for evaluating CAD question answering methods.

Findings

QueryCAD effectively extracts detailed information from CAD models.

The system improves robotic program synthesis by incorporating CAD model understanding.

A new benchmark enables standardized evaluation of CAD question answering methods.

Abstract

CAD models are widely used in industry and are essential for robotic automation processes. However, these models are rarely considered in novel AI-based approaches, such as the automatic synthesis of robot programs, as there are no readily available methods that would allow CAD models to be incorporated for the analysis, interpretation, or extraction of information. To address these limitations, we propose QueryCAD, the first system designed for CAD question answering, enabling the extraction of precise information from CAD models using natural language queries. QueryCAD incorporates SegCAD, an open-vocabulary instance segmentation model we developed to identify and select specific parts of the CAD model based on part descriptions. We further propose a CAD question answering benchmark to evaluate QueryCAD and establish a foundation for future research. Lastly, we integrate QueryCAD within an automatic robot program synthesis framework, validating its ability to enhance deep-learning solutions for robotics by enabling them to process CAD models (https://claudius-kienle.github.com/querycad).