DiagramEval: Evaluating LLM-Generated Diagrams via Graphs

TL;DR

DiagramEval introduces a graph-based evaluation metric for assessing the quality of diagrams generated by large language models, focusing on structure and connections to improve interpretability and accuracy.

Contribution

This paper presents a novel graph-based evaluation metric for LLM-generated diagrams, enabling more accurate and explainable assessment of diagram quality.

Findings

Effective evaluation of LLM-generated diagrams using graph-based metrics

Quantitative validation on recent research literature diagrams

Enhanced explainability provides insights into diagram characteristics

Abstract

Diagrams play a central role in research papers for conveying ideas, yet they are often notoriously complex and labor-intensive to create. Although diagrams are presented as images, standard image generative models struggle to produce clear diagrams with well-defined structure. We argue that a promising direction is to generate demonstration diagrams directly in textual form as SVGs, which can leverage recent advances in large language models (LLMs). However, due to the complexity of components and the multimodal nature of diagrams, sufficiently discriminative and explainable metrics for evaluating the quality of LLM-generated diagrams remain lacking. In this paper, we propose DiagramEval, a novel evaluation metric designed to assess demonstration diagrams generated by LLMs. Specifically, DiagramEval conceptualizes diagrams as graphs, treating text elements as nodes and their connections as directed edges, and evaluates diagram quality using two new groups of metrics: node alignment and path alignment. For the first time, we effectively evaluate diagrams produced by state-of-the-art LLMs on recent research literature, quantitatively demonstrating the validity of our metrics. Furthermore, we show how the enhanced explainability of our proposed metrics offers valuable insights into the characteristics of LLM-generated diagrams. Code: https://github.com/ulab-uiuc/diagram-eval.