SVGEditBench V2: A Benchmark for Instruction-based SVG Editing

TL;DR

SVGEditBench V2 introduces a new benchmark dataset for instruction-based SVG editing, enabling evaluation of current models and highlighting the need for further research in vector graphics editing.

Contribution

The paper presents SVGEditBench V2, a novel dataset for SVG editing tasks created using GPT-4o, facilitating research in instruction-based vector graphics editing.

Findings

Existing LLMs show some success in SVG editing tasks.

The dataset reveals significant room for improvement in current methods.

Varying editing tasks are captured in the dataset.

Abstract

Vector format has been popular for representing icons and sketches. It has also been famous for design purposes. Regarding image editing, research on vector graphics editing rarely exists in contrast with the raster counterpart. We considered the reason to be the lack of datasets and benchmarks. Thus, we propose SVGEditBench V2, a benchmark dataset for instruction-based SVG editing. SVGEditBench V2 comprises triplets of an original image, a ground truth image, and the editing prompt. We built the dataset by first extracting image pairs from various SVG emoji datasets. Then, we had GPT-4o to create the prompt. We found that triplets gained by this simple pipeline contain varying sorts of editing tasks. Additionally, we performed the editing tasks with existing LLMs and investigated how those current methods can perform SVG editing. Although there were some successful cases, we found that there is a massive room for improvement.