SRA: Semantic Relation-Aware Flowchart Question Answering

TL;DR

This paper introduces SRA, a novel approach for FlowchartQA that enhances reasoning by incorporating semantic relations between nodes using LLMs, leading to improved accuracy on the FlowVQA benchmark.

Contribution

It proposes a semantic relation-aware interlanguage for flowchart question answering, enabling deeper reasoning and better handling of complex semantic relationships.

Findings

Significant performance improvements on FlowVQA dataset

Effective integration of semantic relations enhances reasoning depth

Versatile application across different interlanguage formats

Abstract

Flowchart Question Answering (FlowchartQA) is a multi-modal task that automatically answers questions conditioned on graphic flowcharts. Current studies convert flowcharts into interlanguages (e.g., Graphviz) for Question Answering (QA), which effectively bridge modal gaps between questions and flowcharts. More importantly, they reveal the link relations between nodes in the flowchart, facilitating a shallow relation reasoning during tracing answers. However, the existing interlanguages still lose sight of intricate semantic/logic relationships such as Conditional and Causal relations. This hinders the deep reasoning for complex questions. To address the issue, we propose a novel Semantic Relation-Aware (SRA) FlowchartQA approach. It leverages Large Language Model (LLM) to detect the discourse semantic relations between nodes, by which a link-based interlanguage is upgraded to the semantic relation based interlanguage. In addition, we conduct an interlanguage-controllable reasoning process. In this process, the question intention is analyzed with the aim to determine the depth of reasoning (Shallow or Deep reasoning), as well as the well-matched interlanguage. We experiment on the benchmark dataset FlowVQA. The test results show that SRA yields widespread improvements when upgrading different interlanguages like Graphviz, Mermaid and Plantuml