Can Current Agents Close the Discovery-to-Application Gap? A Case Study in Minecraft

TL;DR

This paper introduces SciCrafter, a Minecraft-based benchmark to evaluate AI agents' ability to navigate the entire discovery-to-application loop, revealing current models' limitations and shifting bottlenecks.

Contribution

The paper presents SciCrafter, a novel benchmark for assessing AI's discovery-to-application capabilities, and analyzes the performance and bottlenecks of state-of-the-art models.

Findings

All evaluated models plateau at about 26% success rate.

Knowledge application remains the largest gap across models.

For frontier models, identifying the right problems becomes a major challenge.

Abstract

Discovering causal regularities and applying them to build functional systems--the discovery-to-application loop--is a hallmark of general intelligence, yet evaluating this capacity has been hindered by the vast complexity gap between scientific discovery and real-world engineering. We introduce SciCrafter, a Minecraft-based benchmark that operationalizes this loop through parameterized redstone circuit tasks. Agents must ignite lamps in specified patterns (e.g., simultaneously or in timed sequences); scaling target parameters substantially increases construction complexity and required knowledge, forcing genuine discovery rather than reliance on memorized solutions. Evaluating frontier models including GPT-5.2, Gemini-3-Pro, and Claude-Opus-4.5 under a general-purpose code agent scaffold, we find that all plateau at approximately 26% success rate. To diagnose these failures, we decompose the loop into four capacities--knowledge gap identification, experimental discovery, knowledge consolidation, and knowledge application--and design targeted interventions whose marginal contributions serve as proxies for corresponding gaps. Our analysis reveals that although the general knowledge application capability still remains as the biggest gap across all models, for frontier models the knowledge gap identification starts to become a major hurdle--indicating the bottleneck is shifting from solving problems right to raising the right problems for current AI. We release SciCrafter as a diagnostic probe for future research on AI systems that navigate the full discovery-to-application loop.