SmartOracle -- An Agentic Approach to Mitigate Noise in Differential Oracles

TL;DR

SmartOracle employs specialized LLM-based agents to automate and improve the accuracy of differential oracle validation in JavaScript fuzzing, reducing manual effort and false positives.

Contribution

It introduces an agentic LLM-based architecture for automating oracle validation in differential fuzzing, enhancing accuracy and efficiency.

Findings

Achieves 0.84 recall with 18% false positive rate on benchmarks.

Reduces analysis time by 4× and API costs by 10× compared to baseline.

Successfully identified unknown bugs in major JavaScript engines.

Abstract

Differential fuzzers detect bugs by executing identical inputs across distinct implementations of the same specification, such as JavaScript interpreters. Validating the outputs requires an oracle and for differential testing of JavaScript, these are constructed manually, making them expensive, time-consuming, and prone to false positives. Worse, when the specification evolves, this manual effort must be repeated. Inspired by the success of agentic systems in other SE domains, this paper introduces SmartOracle. SmartOracle decomposes the manual triage workflow into specialized Large Language Model (LLM) sub-agents. These agents synthesize independently gathered evidence from terminal runs and targeted specification queries to reach a final verdict. For historical benchmarks, SmartOracle achieves 0.84 recall with an 18% false positive rate. Compared to a sequential Gemini 2.5 Pro baseline, it improves triage accuracy while reducing analysis time by 4$\times$ and API costs by 10$\times$. In active fuzzing campaigns, SmartOracle successfully identified and reported previously unknown specification-level issues across major engines, including bugs in V8, JavaScriptCore, and GraalJS. The success of SmartOracle's agentic architecture on Javascript suggests it might be useful other software systems- a research direction we will explore in future work.