TOOLCAD: Exploring Tool-Using Large Language Models in Text-to-CAD Generation with Reinforcement Learning

TL;DR

ToolCAD introduces an agentic framework using LLMs for text-to-CAD generation, combining interactive modeling, reinforcement learning, and human supervision to enable open-source LLMs to perform CAD tasks effectively.

Contribution

The paper presents a novel framework for tool-using LLMs in CAD, including an interactive gym and reinforcement learning strategy for improved autonomous modeling.

Findings

ToolCAD enables open-source LLMs to perform CAD tasks comparably to proprietary models.

The framework incorporates hybrid feedback and human supervision for refined model training.

Reinforcement learning enhances the LLM's ability to generate coherent CAD models.

Abstract

Computer-Aided Design (CAD) is an expert-level task that relies on long-horizon reasoning and coherent modeling actions. Large Language Models (LLMs) have shown remarkable advancements in enabling language agents to tackle real-world tasks. Notably, there has been no investigation into how tool-using LLMs optimally interact with CAD engines, hindering the emergence of LLM-based agentic text-to-CAD modeling systems. We propose ToolCAD, a novel agentic CAD framework deploying LLMs as tool-using agents for text-to-CAD generation. Furthermore, we introduce an interactive CAD modeling gym to rollout reasoning and tool-augmented interaction trajectories with the CAD engine, incorporating hybrid feedback and human supervision. Meanwhile, an end-to-end post-training strategy is presented to enable the LLM agent to elicit refined CAD Modeling Chain of Thought (CAD-CoT) and evolve into proficient CAD tool-using agents via online curriculum reinforcement learning. Our findings demonstrate ToolCAD fills the gap in adopting and training open-source LLMs for CAD tool-using agents, enabling them to perform comparably to proprietary models, paving the way for more accessible and robust autonomous text-to-CAD modeling systems.