Using Combinatorial Optimization to Design a High quality LLM Solution

TL;DR

This paper presents a combinatorial optimization-based method for designing high-quality large language model solutions by selecting optimal factor interactions, enabling efficient evaluation and validation of solution pipelines.

Contribution

It introduces a novel approach combining combinatorial optimization with sampling to systematically design and evaluate LLM solutions considering multiple influencing factors.

Findings

Efficiently identifies key factor interactions for LLM solution quality.

Reduces manual effort in designing and evaluating LLM solutions.

Provides a baseline for autoML approaches in LLM solution design.

Abstract

We introduce a novel LLM based solution design approach that utilizes combinatorial optimization and sampling. Specifically, a set of factors that influence the quality of the solution are identified. They typically include factors that represent prompt types, LLM inputs alternatives, and parameters governing the generation and design alternatives. Identifying the factors that govern the LLM solution quality enables the infusion of subject matter expert knowledge. Next, a set of interactions between the factors are defined and combinatorial optimization is used to create a small subset $P$ that ensures all desired interactions occur in $P$. Each element $p \in P$ is then developed into an appropriate benchmark. Applying the alternative solutions on each combination, $p \in P$ and evaluating the results facilitate the design of a high quality LLM solution pipeline. The approach is especially applicable when the design and evaluation of each benchmark in $P$ is time-consuming and involves manual steps and human evaluation. Given its efficiency the approach can also be used as a baseline to compare and validate an autoML approach that searches over the factors governing the solution.