SVGEditBench: A Benchmark Dataset for Quantitative Assessment of LLM's SVG Editing Capabilities

TL;DR

This paper introduces SVGEditBench, a benchmark dataset designed to quantitatively evaluate the ability of large language models to edit SVG vector graphics code, demonstrating GPT-4's superior performance over GPT-3.5.

Contribution

The paper presents SVGEditBench, the first benchmark dataset for assessing LLMs' SVG editing capabilities, along with experimental results comparing GPT-4 and GPT-3.5.

Findings

GPT-4 outperforms GPT-3.5 in SVG editing tasks

SVGEditBench enables quantitative assessment of LLMs' SVG editing skills

The dataset is publicly available for further research

Abstract

Text-to-image models have shown progress in recent years. Along with this progress, generating vector graphics from text has also advanced. SVG is a popular format for vector graphics, and SVG represents a scene with XML text. Therefore, Large Language Models can directly process SVG code. Taking this into account, we focused on editing SVG with LLMs. For quantitative evaluation of LLMs' ability to edit SVG, we propose SVGEditBench. SVGEditBench is a benchmark for assessing the LLMs' ability to edit SVG code. We also show the GPT-4 and GPT-3.5 results when evaluated on the proposed benchmark. In the experiments, GPT-4 showed superior performance to GPT-3.5 both quantitatively and qualitatively. The dataset is available at https://github.com/mti-lab/SVGEditBench.