CABENCH: Benchmarking Composable AI for Solving Complex Tasks through Composing Ready-to-Use Models

TL;DR

CABENCH is a comprehensive benchmark for evaluating composable AI systems that assemble pre-trained models to solve complex tasks, highlighting current capabilities and future challenges.

Contribution

This paper introduces the first public benchmark for composable AI, including a diverse set of tasks, models, and an evaluation framework for systematic assessment.

Findings

Composable AI shows promise in solving complex real-world problems.

Current approaches are outperformed by human-designed solutions in some tasks.

Automating the generation of execution pipelines remains a key challenge.

Abstract

Composable AI offers a scalable and effective paradigm for tackling complex AI tasks by decomposing them into sub-tasks and solving each sub-task using ready-to-use well-trained models. However, systematically evaluating methods under this setting remains largely unexplored. In this paper, we introduce CABENCH, the first public benchmark comprising 70 realistic composable AI tasks, along with a curated pool of 700 models across multiple modalities and domains. We also propose an evaluation framework to enable end-to-end assessment of composable AI solutions. To establish initial baselines, we provide human-designed reference solutions and compare their performance with two LLM-based approaches. Our results illustrate the promise of composable AI in addressing complex real-world problems while highlighting the need for methods that can fully unlock its potential by automatically generating effective execution pipelines.