Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E. Kelly Buchanan, Junhong Shen, Guanghao Ye, Haowei Lin, Jason Poulos, Maoyu Wang, Marianna Nezhurina, Jenia Jitsev, Di Lu
TL;DR
Terminal-Bench 2.0 is a new challenging benchmark with 89 real-world inspired terminal tasks designed to evaluate AI agents' ability to perform complex, long-horizon command line operations, revealing current model limitations.
Contribution
We introduce Terminal-Bench 2.0, a comprehensive and difficult benchmark for AI agents in command line environments, including a dataset, evaluation tools, and analysis of current model performance.
Findings
Frontier models score below 65% on the benchmark
Error analysis highlights key areas for model improvement
Benchmark facilitates future research in real-world terminal tasks
Abstract
AI agents may soon become capable of autonomously completing valuable, long-horizon tasks in diverse domains. Current benchmarks either do not measure real-world tasks, or are not sufficiently difficult to meaningfully measure frontier models. To this end, we present Terminal-Bench 2.0: a carefully curated hard benchmark composed of 89 tasks in computer terminal environments inspired by problems from real workflows. Each task features a unique environment, human-written solution, and comprehensive tests for verification. We show that frontier models and agents score less than 65\% on the benchmark and conduct an error analysis to identify areas for model and agent improvement. We publish the dataset and evaluation harness to assist developers and researchers in future work at https://www.tbench.ai/ .
Peer Reviews
Decision·ICLR 2026 Poster
1. Thorough testing and verification process throughout task development, including algorithmic, LLM-powered, and human review. 1. Builds on standards/checklists from prior work, like MAST and ABC 1. Used a collaborative crowd sourcing process to build a relatively large number of diverse and difficulty tasks 1. Developing a simple, shared scaffold as a reasonable point of comparison 1. Detailed error analysis
1. It's possible that the comparisons would be more fair if the best/preferred scaffold for each agent were used rather than a shared scaffold (which might still favor one model over others). 1. An LLM judge for error analysis was chosen based on agreement with a human annotator on 20 traces, but there are 74 tasks. I think this leaves open a large possibility of sampling bias. 1. "Most agents attempt tasks for less than 20 minutes." This seems very limiting, when most of the tasks are estimated
1. **Comprehensive task coverage**: The benchmark spans multiple levels from basic file operations to complex system configurations, offering good representativeness of real-world scenarios 2. **Automated evaluation pipeline**: Establishes a fairly complete automated testing and scoring infrastructure, reducing subjectivity in assessment 3. **Thorough empirical validation**: Systematic comparison across multiple mainstream models effectively reveals current technological limitations 4. **Strong
1. **Limited diversity in certain categories**: While covering multiple task types, some categories (network configuration, distributed system management) have sparse representation, potentially failing to capture the full spectrum of real-world usage 2. **Narrow evaluation metrics**: Predominantly relies on binary pass/fail assessment, lacking consideration of execution efficiency, code quality, and other important dimensions. The scoring mechanism for partial completion is insufficiently granu
Overall, the paper is exceptionally strong: * It is well-presented, with clear information about the tasks, the task creation process, and agent testing methodology. * The tasks are well-reviewed, lending confidence that the benchmark is of high quality. * A large number of agents were tested * Failure analysis provides valuable information.
The paper would have been stronger if: * we had human baselines (i.e. an expert or junior engineer was asked to complete the task in the same conditions), instead of time estimates as these might over/under estimate the actual time required (famously this is quite hard to accurately do).
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsExplainable Artificial Intelligence (XAI) · Multimodal Machine Learning Applications · Adversarial Robustness in Machine Learning