LLM Code Customization with Visual Results: A Benchmark on TikZ

TL;DR

This paper introduces vTikZ, a benchmark for evaluating how well Large Language Models can customize code to produce specific visual results, revealing current limitations and guiding future research.

Contribution

The paper presents vTikZ, the first benchmark for assessing LLMs' ability to modify code for visual outcomes, including curated scenarios and a visual feedback review tool.

Findings

State-of-the-art LLMs struggle with visual-aligned code modifications.

Current AI code editing methods have significant reliability gaps.

vTikZ enables new research in visual feedback-driven code customization.

Abstract

With the rise of AI-based code generation, customizing existing code out of natural language instructions to modify visual results -such as figures or images -has become possible, promising to reduce the need for deep programming expertise. However, even experienced developers can struggle with this task, as it requires identifying relevant code regions (feature location), generating valid code variants, and ensuring the modifications reliably align with user intent. In this paper, we introduce vTikZ, the first benchmark designed to evaluate the ability of Large Language Models (LLMs) to customize code while preserving coherent visual outcomes. Our benchmark consists of carefully curated vTikZ editing scenarios, parameterized ground truths, and a reviewing tool that leverages visual feedback to assess correctness. Empirical evaluation with stateof-the-art LLMs shows that existing solutions struggle to reliably modify code in alignment with visual intent, highlighting a gap in current AI-assisted code editing approaches. We argue that vTikZ opens new research directions for integrating LLMs with visual feedback mechanisms to improve code customization tasks in various domains beyond TikZ, including image processing, art creation, Web design, and 3D modeling.