The Midas Touch: Triggering the Capability of LLMs for RM-API Misuse Detection

TL;DR

This paper introduces ChatDetector, an LLM-based system that automates the detection of RM-API misuse by understanding documentation, extracting constraints, and accurately pairing APIs, significantly improving detection precision and coverage.

Contribution

The paper presents ChatDetector, a novel LLM-empowered approach that automates RM-API misuse detection through documentation understanding, constraint retrieval, and API pairing, outperforming existing methods.

Findings

Achieves 98.21% precision in RM-API pairing.

Retrieves 47% more RM sentences and 80.85% more constraints than benchmarks.

Identifies 115 security bugs in real applications.

Abstract

In this paper, we propose an LLM-empowered RM-API misuse detection solution, ChatDetector, which fully automates LLMs for documentation understanding which helps RM-API constraints retrieval and RM-API misuse detection. To correctly retrieve the RM-API constraints, ChatDetector is inspired by the ReAct framework which is optimized based on Chain-of-Thought (CoT) to decompose the complex task into allocation APIs identification, RM-object (allocated/released by RM APIs) extraction and RM-APIs pairing (RM APIs usually exist in pairs). It first verifies the semantics of allocation APIs based on the retrieved RM sentences from API documentation through LLMs. Inspired by the LLMs' performance on various prompting methods,ChatDetector adopts a two-dimensional prompting approach for cross-validation. At the same time, an inconsistency-checking approach between the LLMs' output and the reasoning process is adopted for the allocation APIs confirmation with an off-the-shelf Natural Language Processing (NLP) tool. To accurately pair the RM-APIs, ChatDetector decomposes the task again and identifies the RM-object type first, with which it can then accurately pair the releasing APIs and further construct the RM-API constraints for misuse detection. With the diminished hallucinations, ChatDetector identifies 165 pairs of RM-APIs with a precision of 98.21% compared with the state-of-the-art API detectors. By employing a static detector CodeQL, we ethically report 115 security bugs on the applications integrating on six popular libraries to the developers, which may result in severe issues, such as Denial-of-Services (DoS) and memory corruption. Compared with the end-to-end benchmark method, the result shows that ChatDetector can retrieve at least 47% more RM sentences and 80.85% more RM-API constraints.