ReCAD: Reinforcement Learning Enhanced Parametric CAD Model Generation with Vision-Language Models

TL;DR

ReCAD introduces a reinforcement learning framework that leverages pretrained vision-language models to generate accurate, editable parametric CAD models from multimodal inputs, surpassing previous methods in geometric accuracy.

Contribution

The paper presents a novel RL approach that enhances PLMs for precise CAD model generation, incorporating hierarchical primitive learning and a new training strategy.

Findings

Sets new state-of-the-art in text-to-CAD and image-to-CAD tasks.

Significantly reduces Chamfer Distance in image-to-CAD tasks.

Improves geometric accuracy in both in-distribution and out-of-distribution settings.

Abstract

We present ReCAD, a reinforcement learning (RL) framework that bootstraps pretrained large models (PLMs) to generate precise parametric computer-aided design (CAD) models from multimodal inputs by leveraging their inherent generative capabilities. With just access to simple functional interfaces (e.g., point coordinates), our approach enables the emergence of complex CAD operations (e.g., pattern replication and mirror). This stands in contrast to previous methods, which typically rely on knowledge injected through supervised fine-tuning (SFT), offer limited support for editability, and fail to exploit the strong generative priors of PLMs. Specifically, the ReCAD framework begins by fine-tuning vision-language models (VLMs) to equip them with basic CAD model generation capabilities, where we rewrite CAD scripts into parameterized code that is leveraged to generate accurate textual descriptions for supervision. Then, we propose a novel RL strategy that incorporates parameterized code as guidance to enhance the model's reasoning on challenging questions. Furthermore, we employ a hierarchical primitive learning process to progressively teach structured and compositional skills under a unified reward function that ensures both geometric accuracy and semantic fidelity. ReCAD sets a new state-of-the-art in both text-to-CAD and image-to-CAD tasks, significantly improving geometric accuracy across in-distribution and out-of-distribution settings. In the image-to-CAD task, for instance, it reduces the mean Chamfer Distance from 73.47 to 29.61 (in-distribution) and from 272.06 to 80.23 (out-of-distribution), outperforming existing baselines by a substantial margin.