Helix: Evolutionary Reinforcement Learning for Open-Ended Scientific Problem Solving

TL;DR

HELIX is a hierarchical reinforcement learning framework that enhances exploration and solution quality in open-ended scientific problems, leveraging in-context learning and iterative policy refinement.

Contribution

It introduces a novel combination of diverse candidate pools and reinforcement learning for iterative improvement, advancing open-ended problem solving with LLMs.

Findings

Achieved state-of-the-art circle packing results with a 14B model.

Surpassed GPT-4o on standard ML benchmarks with a 5.95 F1 point improvement.

Demonstrated effectiveness in complex scientific problem solving.

Abstract

Large language models (LLMs) with reasoning abilities have demonstrated growing promise for tackling complex scientific problems. Yet such tasks are inherently domain-specific, unbounded and open-ended, demanding exploration across vast and flexible solution spaces. Existing approaches, whether purely learning-based or reliant on carefully designed workflows, often suffer from limited exploration efficiency and poor generalization. To overcome these challenges, we present HELIX -- a Hierarchical Evolutionary reinforcement Learning framework with In-context eXperiences. HELIX introduces two key novelties: (i) a diverse yet high-quality pool of candidate solutions that broadens exploration through in-context learning, and (ii) reinforcement learning for iterative policy refinement that progressively elevates solution quality. This synergy enables the discovery of more advanced solutions. On the circle packing task, HELIX achieves state-of-the-art result with a sum of radii of 2.63598308 using only a 14B model. Across standard machine learning benchmarks, HELIX further surpasses GPT-4o with a carefully engineered pipeline, delivering an average F1 improvement of 5.95 points on the Adult and Bank Marketing datasets.