DeTikZify: Synthesizing Graphics Programs for Scientific Figures and Sketches with TikZ

TL;DR

DeTikZify is a multimodal model that automatically converts sketches and figures into high-quality TikZ graphics, significantly easing the creation of scientific figures by leveraging new datasets and an iterative refinement algorithm.

Contribution

We introduce DeTikZify, a novel model with datasets and an MCTS-based refinement method for converting sketches and figures into semantic TikZ graphics.

Findings

DeTikZify outperforms Claude 3 and GPT-4V in TikZ synthesis.

The MCTS algorithm improves output quality through iterative refinement.

We provide large datasets and code publicly for further research.

Abstract

Creating high-quality scientific figures can be time-consuming and challenging, even though sketching ideas on paper is relatively easy. Furthermore, recreating existing figures that are not stored in formats preserving semantic information is equally complex. To tackle this problem, we introduce DeTikZify, a novel multimodal language model that automatically synthesizes scientific figures as semantics-preserving TikZ graphics programs based on sketches and existing figures. To achieve this, we create three new datasets: DaTikZv2, the largest TikZ dataset to date, containing over 360k human-created TikZ graphics; SketchFig, a dataset that pairs hand-drawn sketches with their corresponding scientific figures; and MetaFig, a collection of diverse scientific figures and associated metadata. We train DeTikZify on MetaFig and DaTikZv2, along with synthetically generated sketches learned from SketchFig. We also introduce an MCTS-based inference algorithm that enables DeTikZify to iteratively refine its outputs without the need for additional training. Through both automatic and human evaluation, we demonstrate that DeTikZify outperforms commercial Claude 3 and GPT-4V in synthesizing TikZ programs, with the MCTS algorithm effectively boosting its performance. We make our code, models, and datasets publicly available.