Trust by Design: Skill Profiles for Transparent, Cost-Aware LLM Routing

TL;DR

BELLA is a framework that helps LLM practitioners select the most suitable model for a task by analyzing required skills, clustering capabilities, and optimizing for cost and performance with transparent recommendations.

Contribution

The paper introduces BELLA, a novel skill-based, interpretable framework for cost-aware LLM model selection that improves transparency and task-specific performance.

Findings

BELLA effectively decomposes LLM outputs into skills.

It clusters skills into structured capability matrices.

The framework enables principled cost-performance trade-offs.

Abstract

How should Large Language Model (LLM) practitioners select the right model for a task without wasting money? We introduce BELLA (Budget-Efficient LLM Selection via Automated skill-profiling), a framework that recommends optimal LLM selection for tasks through interpretable skill-based model selection. Standard benchmarks report aggregate metrics that obscure which specific capabilities a task requires and whether a cheaper model could suffice. BELLA addresses this gap through three stages: (1) decomposing LLM outputs and extract the granular skills required by using critic-based profiling, (2) clustering skills into structured capability matrices, and (3) multi-objective optimization to select the right models to maximize performance while respecting budget constraints. BELLA provides natural-language rationale for recommendations, providing transparency that current black-box routing systems lack. We describe the framework architecture, situate it within the landscape of LLM routing and evaluation, and discuss its application to financial reasoning as a representative domain exhibiting diverse skill requirements and cost-variation across models. Our framework enables practitioners to make principled and cost-performance trade-offs for deploying LLMs.