Text-to-CAD Evaluation with CADTests

TL;DR

This paper introduces CADTestBench, a novel automated testing benchmark for evaluating Text-to-CAD models, enabling comprehensive assessment and guiding model improvement.

Contribution

It presents CADTestBench, the first test-based benchmark for Text-to-CAD, and demonstrates its effectiveness in evaluation and guiding model development.

Findings

CADTestBench enables comprehensive benchmarking of Text-to-CAD methods.

Using CADTests can improve CAD model generation performance.

Baseline methods guided by CADTests outperform current approaches.

Abstract

Text-to-CAD has recently emerged as an important task with the potential to substantially accelerate design workflows. Despite its significance, there has been surprisingly little work on Text-to-CAD evaluation, and assessing CAD model generation performance remains a considerable challenge. In this work, we introduce a new evaluation perspective for Text-to-CAD based on automated testing. We propose CADTestBench, the first test-based benchmark for Text-to-CAD, based on CADTests, executable software tests that verify whether a generated CAD model satisfies the geometric and topological requirements of the input prompt. Using CADTestBench, we conduct comprehensive benchmarking of recent Text-to-CAD methods and further demonstrate that CADTests can also guide CAD model generation, yielding simple baselines that surpass performance of current methods. CADTestBench code and data are available at GitHub and Hugging Face dataset.