Column Generation for Interaction Coverage in Combinatorial Software Testing

TL;DR

This paper introduces a hybrid column generation framework combining Mathematical Programming and Constraint Programming to efficiently generate combinatorial test cases with coverage guarantees, significantly reducing testing effort.

Contribution

It presents a novel, generic column generation algorithm for combinatorial testing that supports heterogeneous alphabets and is implemented as a practical cloud service.

Findings

Algorithm is recognized as a top cloud application at Oracle.

Service aids diverse software testing teams.

Reduces number of tests compared to exhaustive methods.

Abstract

This paper proposes a novel column generation framework for combinatorial software testing. In particular, it combines Mathematical Programming and Constraint Programming in a hybrid decomposition to generate covering arrays. The approach allows generating parameterized test cases with coverage guarantees between parameter interactions of a given application. Compared to exhaustive testing, combinatorial test case generation reduces the number of tests to run significantly. Our column generation algorithm is generic and can accommodate mixed coverage arrays over heterogeneous alphabets. The algorithm is realized in practice as a cloud service and recognized as one of the five winners of the company-wide cloud application challenge at Oracle. The service is currently helping software developers from a range of different product teams in their testing efforts while exposing declarative constraint models and hybrid optimization techniques to a broader audience.