From Natural Language to Executable Properties for Property-based Testing of Mobile Apps

TL;DR

This paper introduces iPBT, a tool that automatically converts natural language descriptions into executable properties for property-based testing of mobile apps, significantly reducing manual effort and improving robustness.

Contribution

It presents a novel structured property synthesis approach using multimodal LLMs and in-context learning, enabling automatic translation of natural language into executable testing properties.

Findings

iPBT achieves 95.2% accuracy on property synthesis.

Enriched widget context improves accuracy by up to 20.2%.

Reduces manual effort by 56% in writing executable properties.

Abstract

Property-based testing (PBT) is a popular software testing methodology and is effective in validating the functionality of mobile applications (apps for short). However, its adoption in practice remains limited, largely due to the manual effort and technical expertise required to specify executable properties. In this experience paper, we propose a novel structured property synthesis approach that automatically translates property descriptions in natural language into executable properties, and implement it in a tool named iPBT. Our approach decomposes the problem into UI semantic grounding and executable property synthesis. It first builds an enriched widget context via multimodal LLMs to align visual elements with their functional semantics, and then uses an LLM with in-context learning to generate framework-specific executable properties. We evaluate iPBT with a closed-source LLM (GPT-4o) and an open-source LLM (DeepSeek-V3) on 124 diverse property descriptions derived from an existing benchmark dataset. iPBT achieves 95.2% (118/124) accuracy on both LLMs. Notably, an ablation study reveals that the enriched widget context contributes to an absolute improvement of up to 20.2% (from 75.0% to 95.2%). A user study with 10 participants demonstrates that iPBT reduces the time required to write executable properties by 56%, suggesting substantially lower manual effort. Furthermore, evaluations on 1,180 linguistically diverse variations demonstrate iPBT's robustness (87.6% accuracy), indicating its capability to handle varied expressions.