Search-Based Risk Feature Discovery in Document Structure Spaces under a Constrained Budget

TL;DR

This paper introduces a search-based testing approach to uncover diverse failure modes in document processing systems within limited budgets, emphasizing solver complementarity for robust validation.

Contribution

It formalizes risk feature discovery as a search problem and benchmarks multiple search strategies, highlighting the importance of solver diversity in failure detection.

Findings

Different solvers find unique failure modes at similar budgets.

No single search strategy dominates across all budgets.

Combining solvers improves failure coverage and robustness.

Abstract

Enterprise-grade Intelligent Document Processing (IDP) systems support high-stakes workflows across finance, insurance, and healthcare. Early-phase system validation under limited budgets mandates uncovering diverse failure mechanisms, rather than identifying a single worst-case document. We formalize this challenge as a Search-Based Software Testing (SBST) problem, aiming to identify complex interactions between document variables, with the objective to maximize the number of distinct failure types discovered within a fixed evaluation budget. Our methodology operates on a combinatorial space of document configurations, rendering instances of structural \emph{risk features} to induce realistic failure conditions. We benchmark a diverse portfolio of search strategies spanning evolutionary, swarm-based, quality-diversity, learning-based, and quantum under identical budget constraints. Through configuration-level exclusivity, win-rate, and cross-temporal overlap analyses, we show that different solvers consistently uncover failure modes that remain undiscovered by specific alternatives at comparable budgets. Crucially, cross-temporal analysis reveals persistent solver-specific discoveries across all evaluated budgets, with no single strategy exhibiting absolute dominance. While the union of all solvers eventually recovers the observed failure space, reliance on any individual method systematically delays the discovery of important risks. These results demonstrate intrinsic solver complementarity and motivate portfolio-based SBST strategies for robust industrial IDP validation.