Combination generators with optimal cache utilization and communication free parallel execution

TL;DR

This paper presents a novel combination generator that achieves optimal cache utilization, constant amortized time, and parallel execution, suitable for exhaustive and combinatorial optimization tasks, with formal correctness guarantees.

Contribution

It introduces a new recursive, efficient combination generator with optimal properties and extends the approach to nested structures, a novel contribution in the field.

Findings

Achieves constant amortized time for combination generation.

Ensures optimal cache utilization and parallelizability.

Extends to nested permutations and combinations with Gray code orderings.

Abstract

We introduce an efficient and elegant combination generator for producing all combinations of size less than or equal to K, designed for exhaustive generation and combinatorial optimization tasks. This generator can be implemented to achieve what we define as optimal efficiency: constant amortized time, optimal cache utilization, embarrassingly parallel execution, and a recursive structure compatible with pruning-based search. These properties are difficult to satisfy simultaneously in existing generators. For example, classical Gray code or lexicographic generators are typically list-based and sequentially defined, making them difficult to vectorized, inefficient in cache usage, and inherently hard to parallelize. Generators based on unranking methods, while easy to parallelize, are non-recursive. These limitations reduce their applicability in our target applications, where both computational efficiency and recursion are crucial. We adapt Bird's algebra of programming-style calculation to derive our algorithms, a formalism for developing correct-by-construction programs from specifications. As a result, all generators in this paper are first formulated in their clearest specification, and efficient definitions are derived constructively through equational reasoning, resulting in concise and elegant divide-and-conquer definitions. Beyond presenting a combination generator, we extend our approach to construct generators for K-permutations, nested combinations of combinations, and nested permutation-combination structures. To the best of our knowledge, the literature has not previously reported generators for these nested structures. We also develop sequential variants that produce configurations in Gray code-compatible orders -- such as the revolving door ordering -- which are particularly useful for constructing nested generators.