DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research

TL;DR

This paper introduces RLER, a reinforcement learning method with evolving rubrics, enabling deep research agents to produce high-quality long-form answers, outperforming existing models in multiple domains.

Contribution

It presents RLER, a novel reinforcement learning approach that co-evolves rubrics with the policy model for improved long-form research capabilities.

Findings

DR Tulu-8B outperforms existing open deep research agents by 15.6%.

DR Tulu-8B matches or exceeds proprietary agents by 0.7%.

It is 1000x cheaper per query than OpenAI DR.

Abstract

Deep research agents perform multi-step research to produce long-form, well-attributed answers. However, most open deep research agents are trained on easily verifiable short-form QA tasks via reinforcement learning with verifiable rewards, which does not extend to realistic long-form tasks. We address this with Reinforcement Learning with Evolving Rubrics (RLER), where rubrics are constructed and maintained to co-evolve with the policy model during training. This allows the rubrics to incorporate newly explored information from search and contrasting model responses, enabling better fact checking and more discriminative on-policy feedback. Using RLER, we develop Deep Research Tulu (DR Tulu-8B), the first fully open model that is directly trained for open-ended, long-form deep research. Across four long-form deep research benchmarks in science, healthcare, and general domains, DR Tulu substantially outperforms existing open deep research agents (by 15.6% over Tongyi DR on average) and matches or exceeds proprietary deep research agents (by 0.7% over OpenAI DR on average), while being significantly smaller and cheaper per query (1000x cheaper than OpenAI DR per query).