AppSelectBench: Application-Level Tool Selection Benchmark

TL;DR

AppSelectBench is a comprehensive benchmark designed to evaluate application-level reasoning in computer-using agents, addressing a gap in existing tools by focusing on selecting the appropriate application based on realistic user intents.

Contribution

It introduces a large-scale, diverse, and realistic benchmark with evaluation protocols for assessing application selection capabilities in CUAs, including a novel user task generation pipeline.

Findings

Models show systematic weaknesses in inter-application reasoning.

Even the most capable models struggle with consistent application choices.

The benchmark reveals strengths and weaknesses of large language models in application reasoning.

Abstract

Computer Using Agents (CUAs) are increasingly equipped with external tools, enabling them to perform complex and realistic tasks. For CUAs to operate effectively, application selection, which refers to deciding which application to use before invoking fine-grained tools such as APIs, is a fundamental capability. It determines whether the agent initializes the correct environment, avoids orchestration confusion, and efficiently focuses on relevant context. However, existing benchmarks primarily assess fine-grained API selection, offering limited insight into whether models can reason across and choose between different applications. To fill this gap, we introduce AppSelectBench, a comprehensive benchmark for evaluating application selection in CUAs. AppSelectBench contains a novel user task generation pipeline that produces realistic, diverse, and semantically grounded user intents at scale, together with unified evaluation protocols covering random, heuristic, zero-shot, few-shot, and retrieval-augmented-settings. AppSelectBench covers one hundred widely used desktop applications and includes more than one hundred thousand realistic, diverse, and semantically grounded user tasks. Extensive experiments across both closed-source and open-source large language models reveal systematic strengths and weaknesses in inter-application reasoning, showing that even the most capable models still struggle to make consistent application choices. Together, these results establish AppSelectBench as a foundation for studying and advancing application level reasoning, an essential yet underexplored capability of intelligent CUAs. The source is available at https://microsoft.github.io/appselectbench/.