Contrastive Concept-Tree Search for LLM-Assisted Algorithm Discovery

TL;DR

This paper introduces Contrastive Concept-Tree Search (CCTS), a novel method that leverages hierarchical concept representations and contrastive learning to enhance LLM-assisted algorithm discovery, improving search efficiency and interpretability.

Contribution

CCTS extracts and utilizes a hierarchical concept model with contrastive reweighting to guide program search, representing a new approach to exploiting LLM internal representations.

Findings

CCTS outperforms fitness-based baselines in search efficiency.

CCTS produces interpretable, task-specific concept trees.

Learning which concepts to avoid drives the performance gains.

Abstract

Large language Model (LLM)-assisted algorithm discovery is an iterative, black-box optimization process over programs to approximatively solve a target task, where an LLM proposes candidate programs and an external evaluator provides task feedback. Despite intense recent research on the topic and promising results, how can the LLM internal representation of the space of possible programs be maximally exploited to improve performance is an open question. Here, we introduce Contrastive Concept-Tree Search (CCTS), which extracts a hierarchical concept representation from the generated programs and learns a contrastive concept model that guides parent selection. By reweighting parents using a likelihood-ratio score between high- and low-performing solutions, CCTS biases search toward useful concept combinations and away from misleading ones, providing guidance through an explicit concept hierarchy rather than the algorithm lineage constructed by the LLM. We show that CCTS improves search efficiency over fitness-based baselines and produces interpretable, task-specific concept trees across a benchmark of open Erd\H{o}s-type combinatorics problems. Our analysis indicates that the gains are driven largely by learning which concepts to avoid. We further validate these findings in a controlled synthetic algorithm-discovery environment, which reproduces qualitatively the search dynamics observed with the LLMs.