JFTA-Bench: Evaluate LLM's Ability of Tracking and Analyzing Malfunctions Using Fault Trees

TL;DR

This paper introduces JFTA-Bench, a benchmark for evaluating large language models' ability to interpret fault trees in maintenance tasks, focusing on malfunction tracking and analysis in complex systems.

Contribution

It proposes a novel textual representation for fault trees and constructs a comprehensive benchmark to assess LLMs in malfunction localization and error recovery tasks.

Findings

Gemini 2.5 pro achieves the best performance on the benchmark.

The benchmark contains 3,130 entries with an average of 40.75 dialogue turns.

The model demonstrates capabilities in task tracking and error recovery in complex environments.

Abstract

In the maintenance of complex systems, fault trees are used to locate problems and provide targeted solutions. To enable fault trees stored as images to be directly processed by large language models, which can assist in tracking and analyzing malfunctions, we propose a novel textual representation of fault trees. Building on it, we construct a benchmark for multi-turn dialogue systems that emphasizes robust interaction in complex environments, evaluating a model's ability to assist in malfunction localization, which contains $3130$ entries and $40.75$ turns per entry on average. We train an end-to-end model to generate vague information to reflect user behavior and introduce long-range rollback and recovery procedures to simulate user error scenarios, enabling assessment of a model's integrated capabilities in task tracking and error recovery, and Gemini 2.5 pro archives the best performance.