An Online Reference-Free Evaluation Framework for Flowchart Image-to-Code Generation

TL;DR

This paper introduces a reference-free, inference-time evaluation framework for flowchart image-to-code generation that uses OCR and Visual Entailment to assess quality without ground-truth references.

Contribution

It proposes two automated metrics, Recall OCR and Precision VE, combined into an F1 score, enabling reliable quality assessment in real-world, production environments.

Findings

High correlation with ground-truth metrics (r > 0.9)

Effective in continuous quality monitoring

Applicable to arbitrary flowchart inputs

Abstract

Vision-Language Models (VLMs) are increasingly used in document processing pipelines to convert flowchart images into structured code (e.g., Mermaid). In production, these systems process arbitrary inputs for which no ground-truth code exists, making output quality difficult to assess. We propose a reference-free evaluation framework that monitors flowchart image-to-code generation quality at inference time, using only the input image and the generated output. The framework introduces two automated metrics: $\text{Recall}{\text{OCR}}$, which estimates content coverage by extracting text from the input image via OCR as a proxy reference, and $\text{Precision}{\text{VE}}$, which detects hallucinated elements through Visual Entailment against the original image. Their harmonic mean, $\text{F1}{\text{OCR-VE}}$, provides a unified quality score. Validation on the FlowVQA dataset shows strong agreement with ground-truth metrics (average Pearson's $r = 0.97$, $0.91$, and $0.94$ for Recall, Precision, and F1, respectively), confirming the framework's reliability as a practical, reference-free alternative for continuous quality monitoring in production settings.