Diagnosing Robotics Systems Issues with Large Language Models

TL;DR

This paper explores the use of large language models for diagnosing issues in robotics systems, demonstrating that fine-tuned models can outperform larger models like GPT-4 in accuracy and cost-effectiveness.

Contribution

It introduces SYSDIAGBENCH, a new robotics diagnostics benchmark, and shows that fine-tuned LLMs can effectively perform root cause analysis in robotics.

Findings

QLoRA fine-tuning enables small models to outperform GPT-4 in diagnostics

A 7B-parameter model achieves accuracy comparable to GPT-4

Cost-effective LLMs can match human expert approval ratings

Abstract

Quickly resolving issues reported in industrial applications is crucial to minimize economic impact. However, the required data analysis makes diagnosing the underlying root causes a challenging and time-consuming task, even for experts. In contrast, large language models (LLMs) excel at analyzing large amounts of data. Indeed, prior work in AI-Ops demonstrates their effectiveness in analyzing IT systems. Here, we extend this work to the challenging and largely unexplored domain of robotics systems. To this end, we create SYSDIAGBENCH, a proprietary system diagnostics benchmark for robotics, containing over 2500 reported issues. We leverage SYSDIAGBENCH to investigate the performance of LLMs for root cause analysis, considering a range of model sizes and adaptation techniques. Our results show that QLoRA finetuning can be sufficient to let a 7B-parameter model outperform GPT-4 in terms of diagnostic accuracy while being significantly more cost-effective. We validate our LLM-as-a-judge results with a human expert study and find that our best model achieves similar approval ratings as our reference labels.