Plug-and-Play Performance Estimation for LLM Services without Relying on Labeled Data

TL;DR

This paper presents a plug-and-play method to estimate LLM service performance across tasks using only unlabeled samples, leveraging negative log-likelihood and perplexity as key features, without relying on labeled data.

Contribution

It introduces a novel performance estimation approach for LLM services that requires no labeled data and can be applied directly during service invocation.

Findings

Effective performance estimation using NLL and perplexity features

Comparison shows superiority over baseline methods

Demonstrated applicability in service selection and optimization

Abstract

Large Language Model (LLM) services exhibit impressive capability on unlearned tasks leveraging only a few examples by in-context learning (ICL). However, the success of ICL varies depending on the task and context, leading to heterogeneous service quality. Directly estimating the performance of LLM services at each invocation can be laborious, especially requiring abundant labeled data or internal information within the LLM. This paper introduces a novel method to estimate the performance of LLM services across different tasks and contexts, which can be "plug-and-play" utilizing only a few unlabeled samples like ICL. Our findings suggest that the negative log-likelihood and perplexity derived from LLM service invocation can function as effective and significant features. Based on these features, we utilize four distinct meta-models to estimate the performance of LLM services. Our proposed method is compared against unlabeled estimation baselines across multiple LLM services and tasks. And it is experimentally applied to two scenarios, demonstrating its effectiveness in the selection and further optimization of LLM services.