COSMosFL: Ensemble of Small Language Models for Fault Localisation

TL;DR

COSMos is an ensemble method combining small and large language models for fault localization, balancing accuracy and cost, demonstrated to be effective on Defects4J with Pareto-optimal results.

Contribution

This paper introduces COSMos, a novel ensemble technique for fault localization that leverages both small and large language models to optimize accuracy and efficiency.

Findings

COSMos achieves Pareto-optimal trade-offs between accuracy and inference cost.

Ensemble of SLMs and LLMs improves fault localization performance.

Empirical results on Defects4J validate the effectiveness of COSMos.

Abstract

LLMs are rapidly being adopted to build powerful tools and agents for software engineering, but most of them rely heavily on extremely large closed-source models. This, in turn, can hinder wider adoption due to security issues as well as financial cost and environmental impact. Recently, a number of open source Small Language Models (SLMs) are being released and gaining traction. While SLMs are smaller, more energy-efficient, and therefore easier to locally deploy, they tend to show worse performance when compared to larger closed LLMs. We present COSMos, a task-level LLM ensemble technique that uses voting mechanism, to provide a broader range of choice between SLMs and LLMs. We instantiate COSMos with an LLM-based Fault Localisation technique, AutoFL, and report the cost-benefit trade-off between LLM accuracy and various costs such as energy consumption, inference time, and the number of tokens used. An empirical evaluation using Defects4J shows that COSMos can build effective ensembles that can achieve Pareto-optimality in terms of FL accuracy and inference cost, when compared to individual models.