Agentick: A Unified Benchmark for General Sequential Decision-Making Agents

TL;DR

Agentick is a comprehensive benchmark for evaluating diverse sequential decision-making agents across multiple modalities, tasks, and difficulty levels, facilitating fair comparison and progress in AI research.

Contribution

It introduces a unified, multi-modal benchmark with extensive evaluation tools, enabling fair comparison of RL, LLM, VLM, hybrid, and human agents in sequential decision-making.

Findings

No single approach dominates across all tasks.

GPT-5 mini outperforms others overall.

Reasoning harness significantly boosts LLM performance.

Abstract

AI agent research spans a wide spectrum: from RL agents that learn from scratch to foundation model agents that leverage pre-trained knowledge, yet no unified benchmark enables fair comparison across these approaches. We present Agentick, a benchmark for sequential decision-making agents designed to evaluate RL, LLM, VLM, hybrid, and human agents on common ground and to power research on the fundamental challenges of sequential decision-making. Agentick provides 37 procedurally generated tasks across six capability categories, four difficulty levels, and five observation modalities, all exposed through a single Gymnasium-compatible interface. The benchmark ships with a Coding API, oracle reference policies for all tasks, pre-built SFT datasets, a composable agent harness, and a live leaderboard. An evaluation spanning 27 configurations and over 90,000 episodes reveals that no single approach dominates: GPT-5 mini leads overall at 0.309 oracle-normalized score while PPO dominates planning and multi-agent tasks; the reasoning harness multiplies LLM performance by 3-10x; and ASCII observations consistently outperform natural language. These findings highlight the substantial room for improvement that remains across all agent paradigms. Agentick's capability-decomposed, multi-modal design provides the empirical infrastructure needed to drive progress toward general autonomous agents, both as an evaluation framework and as a training ground for RL post-training of foundation models in truly sequential environments.