CMDBench: A Benchmark for Coarse-to-fine Multimodal Data Discovery in Compound AI Systems

TL;DR

CMDBench is a new benchmark designed to evaluate the effectiveness of multimodal data discovery in Compound AI systems within complex enterprise data environments, addressing a gap in existing benchmarks.

Contribution

It introduces a benchmark modeling enterprise data complexity and adapts existing datasets to evaluate coarse- and fine-grained data retrieval in CASs.

Findings

Data retriever design significantly impacts task accuracy, with an average 46% drop.

Multimodal data sources and task difficulty affect retrieval performance.

Optimization strategies are needed for better LLM agent and retriever selection.

Abstract

Compound AI systems (CASs) that employ LLMs as agents to accomplish knowledge-intensive tasks via interactions with tools and data retrievers have garnered significant interest within database and AI communities. While these systems have the potential to supplement typical analysis workflows of data analysts in enterprise data platforms, unfortunately, CASs are subject to the same data discovery challenges that analysts have encountered over the years -- silos of multimodal data sources, created across teams and departments within an organization, make it difficult to identify appropriate data sources for accomplishing the task at hand. Existing data discovery benchmarks do not model such multimodality and multiplicity of data sources. Moreover, benchmarks of CASs prioritize only evaluating end-to-end task performance. To catalyze research on evaluating the data discovery performance of multimodal data retrievers in CASs within a real-world setting, we propose CMDBench, a benchmark modeling the complexity of enterprise data platforms. We adapt existing datasets and benchmarks in open-domain -- from question answering and complex reasoning tasks to natural language querying over structured data -- to evaluate coarse- and fine-grained data discovery and task execution performance. Our experiments reveal the impact of data retriever design on downstream task performance -- a 46% drop in task accuracy on average -- across various modalities, data sources, and task difficulty. The results indicate the need to develop optimization strategies to identify appropriate LLM agents and retrievers for efficient execution of CASs over enterprise data.