syftr: Pareto-Optimal Generative AI

TL;DR

syftr is a framework that efficiently searches for Pareto-optimal RAG configurations, balancing accuracy and cost, and adapts quickly to new modules, significantly improving the design of generative AI pipelines.

Contribution

syftr introduces a Bayesian optimization-based method with early stopping to find cost-effective, high-accuracy RAG configurations, including agentic modules, in a broad search space.

Findings

syftr finds flows about 9 times cheaper on average.

syftr maintains most accuracy of top-performing flows.

syftr accelerates the design of high-quality generative AI pipelines.

Abstract

Retrieval-Augmented Generation (RAG) pipelines are central to applying large language models (LLMs) to proprietary or dynamic data. However, building effective RAG flows is complex, requiring careful selection among vector databases, embedding models, text splitters, retrievers, and synthesizing LLMs. The challenge deepens with the rise of agentic paradigms. Modules like verifiers, rewriters, and rerankers-each with intricate hyperparameter dependencies have to be carefully tuned. Balancing tradeoffs between latency, accuracy, and cost becomes increasingly difficult in performance-sensitive applications. We introduce syftr, a framework that performs efficient multi-objective search over a broad space of agentic and non-agentic RAG configurations. Using Bayesian Optimization, syftr discovers Pareto-optimal flows that jointly optimize task accuracy and cost. A novel early-stopping mechanism further improves efficiency by pruning clearly suboptimal candidates. Across multiple RAG benchmarks, syftr finds flows which are on average approximately 9 times cheaper while preserving most of the accuracy of the most accurate flows on the Pareto-frontier. Furthermore, syftr's ability to design and optimize allows integrating new modules, making it even easier and faster to realize high-performing generative AI pipelines.