Follow the Flow: Fine-grained Flowchart Attribution with Neurosymbolic Agents

TL;DR

This paper introduces a neurosymbolic agent for fine-grained flowchart attribution, improving the interpretability and reliability of LLMs analyzing complex flowcharts by reducing hallucinations and enhancing explanation accuracy.

Contribution

It proposes FlowPathAgent, a novel graph-based reasoning method for flowchart attribution, and introduces FlowExplainBench, a benchmark for evaluating attribution quality across diverse flowcharts.

Findings

FlowPathAgent reduces hallucinations in flowchart QA by 10-14%.

The approach outperforms strong baselines on the FlowExplainBench dataset.

Enhanced explainability and verifiability of LLM responses to flowcharts.

Abstract

Flowcharts are a critical tool for visualizing decision-making processes. However, their non-linear structure and complex visual-textual relationships make it challenging to interpret them using LLMs, as vision-language models frequently hallucinate nonexistent connections and decision paths when analyzing these diagrams. This leads to compromised reliability for automated flowchart processing in critical domains such as logistics, health, and engineering. We introduce the task of Fine-grained Flowchart Attribution, which traces specific components grounding a flowchart referring LLM response. Flowchart Attribution ensures the verifiability of LLM predictions and improves explainability by linking generated responses to the flowchart's structure. We propose FlowPathAgent, a neurosymbolic agent that performs fine-grained post hoc attribution through graph-based reasoning. It first segments the flowchart, then converts it into a structured symbolic graph, and then employs an agentic approach to dynamically interact with the graph, to generate attribution paths. Additionally, we present FlowExplainBench, a novel benchmark for evaluating flowchart attributions across diverse styles, domains, and question types. Experimental results show that FlowPathAgent mitigates visual hallucinations in LLM answers over flowchart QA, outperforming strong baselines by 10-14% on our proposed FlowExplainBench dataset.