Combating Missed Recalls in E-commerce Search: A CoT-Prompting Testing Approach

TL;DR

This paper introduces mrDetector, an automatic testing approach for e-commerce search systems that effectively identifies missed recalls by generating user-aligned test cases and providing a test oracle, improving detection accuracy.

Contribution

The paper presents mrDetector, the first automated method for detecting missed recalls in e-commerce search, utilizing LLM-based query generation and metamorphic testing for high accuracy.

Findings

Outperforms all baselines with lowest false positive ratio

Discovers over 100 missed recalls with only 17 false positives

Effective in real industrial data environments

Abstract

Search components in e-commerce apps, often complex AI-based systems, are prone to bugs that can lead to missed recalls - situations where items that should be listed in search results aren't. This can frustrate shop owners and harm the app's profitability. However, testing for missed recalls is challenging due to difficulties in generating user-aligned test cases and the absence of oracles. In this paper, we introduce mrDetector, the first automatic testing approach specifically for missed recalls. To tackle the test case generation challenge, we use findings from how users construct queries during searching to create a CoT prompt to generate user-aligned queries by LLM. In addition, we learn from users who create multiple queries for one shop and compare search results, and provide a test oracle through a metamorphic relation. Extensive experiments using open access data demonstrate that mrDetector outperforms all baselines with the lowest false positive ratio. Experiments with real industrial data show that mrDetector discovers over one hundred missed recalls with only 17 false positives.