FlyAOC: Evaluating Agentic Ontology Curation of Drosophila Scientific Knowledge Bases

TL;DR

FlyAOC introduces FlyBench, a comprehensive benchmark for evaluating AI agents in the complex task of curating Drosophila scientific knowledge, emphasizing end-to-end reasoning, retrieval, and annotation from large literature corpora.

Contribution

This paper presents FlyBench, the first benchmark for end-to-end agentic ontology curation from scientific literature, and evaluates multiple agent architectures on this challenging task.

Findings

Multi-agent architectures outperform simpler models.

Scaling backbone models yields diminishing returns.

Agents mainly use retrieval for confirmation, not discovery.

Abstract

Scientific knowledge bases accelerate discovery by curating findings from primary literature into structured, queryable formats for both human researchers and emerging AI systems. Maintaining these resources requires expert curators to search relevant papers, reconcile evidence across documents, and produce ontology-grounded annotations - a workflow that existing benchmarks, focused on isolated subtasks like named entity recognition or relation extraction, do not capture. We present FlyBench to evaluate AI agents on end-to-end agentic ontology curation from scientific literature. Given only a gene symbol, agents must search and read from a corpus of 16,898 full-text papers to produce structured annotations: Gene Ontology terms describing function, expression patterns, and historical synonyms linking decades of nomenclature. The benchmark includes 7,397 expert-curated annotations across 100 genes drawn from FlyBase, the Drosophila (fruit fly) knowledge base. We evaluate four baseline agent architectures: memorization, fixed pipeline, single-agent, and multi-agent. We find that architectural choices significantly impact performance, with multi-agent designs outperforming simpler alternatives, yet scaling backbone models yields diminishing returns. All baselines leave substantial room for improvement. Our analysis surfaces several findings to guide future development; for example, agents primarily use retrieval to confirm parametric knowledge rather than discover new information. We hope FlyBench will drive progress on retrieval-augmented scientific reasoning, a capability with broad applications across scientific domains.