Can Large Language Models Detect Real-World Android Software Compliance Violations?

TL;DR

This paper introduces CompliBench, a new evaluation framework for assessing large language models' ability to detect compliance violations in Android applications across various legal standards, addressing current model limitations.

Contribution

The paper presents CompliBench, a novel benchmarking framework with new tasks and stability-aware metrics for evaluating LLMs' compliance detection capabilities in Android apps.

Findings

Claude-3.5-sonnet-20241022 achieved highest OCS score (0.3295)

CompliBench improves assessment of LLMs in compliance detection

Models show varying performance across legal frameworks

Abstract

The rapid development of Large Language Models (LLMs) has transformed software engineering, showing promise in tasks like code generation, bug detection, and compliance checking. However, current models struggle to detect compliance violations in Android applications across diverse legal frameworks. We propose \emph{CompliBench}, a novel evaluation framework for assessing LLMs' ability to detect compliance violations under regulations like LGPD, PDPA, and PIPEDA. The framework defines two tasks: Task 1 evaluates \emph{retrieval and localization} at file, module, and line granularities, and Task 2 assesses \emph{multi-label judgment} for code snippets. These tasks mirror the audit process, where auditors locate problematic code and determine implicated provisions. Traditional metrics fail to capture important aspects like cross-granularity stability and jurisdictional consistency. Thus, we introduce stability-aware composites (SGS, RCS, CRGS, and OCS) for a more comprehensive assessment. Experiments with six models, including GPT-4O and Claude-3.5, show \emph{CompliBench} improves compliance detection, with Claude-3.5-sonnet-20241022 achieving the highest OCS score (0.3295), and Gemini-2.5-pro the lowest (0.0538). This work demonstrates \emph{CompliBench}'s potential for improving LLM performance in compliance tasks and provides a foundation for future tools aligned with data protection standards. Our project is available at https://github.com/Haoyi-Zhang/CompliBench.