Lachesis: Predicting LLM Inference Accuracy using Structural Properties of Reasoning Paths

TL;DR

Lachesis is a predictive model that uses structural properties of reasoning paths to estimate the correctness of LLM inferences, enabling early stopping and improving reliability in complex reasoning tasks.

Contribution

This paper introduces Lachesis, a novel predictive approach combining LSTM and GCN models to assess inference correctness based on reasoning path properties.

Findings

Lachesis achieves up to 81.36% precision in predicting answer correctness.

The model enables early termination of unpromising inferences.

Empirical evaluation demonstrates effectiveness with AutoFL fault localization.

Abstract

Large Language Models are increasingly used to build agents to perform more complex tasks. As LLMs perform more complicated reasoning through longer interactions, self-consistency, i.e., the idea that the answer obtained from sampling and marginalising a number of multiple independent inferences is more likely to be correct, has received much attention as a simple validation technique. This paper aims to empirically verify this intuitive hypothesis by predicting the correctness of answers obtained using self-consistency from properties of the samples of reasoning paths. We introduce Lachesis, a predictive model for self-consistency based LLM inferences, and empirically evaluate it using AutoFL, a recently proposed LLM-based fault localisation technique, as the target technique that uses self-consistency. Lachesis converts collected reasoning paths from AutoFL using specifically designed reasoning path representations, and trains LSTM and GCN models to predict whether a given set of reasoning paths would result in a correct answer. The results suggest that Lachesis can predict the correctness of answers with a precision of up to 0.8136, highlighting the possibility of training a predictive model that can allow early termination of inferences that are not likely to be successful.