CodeDistiller: Automatically Generating Code Libraries for Scientific Coding Agents

TL;DR

CodeDistiller automatically creates vetted, domain-specific code libraries from GitHub repositories to enhance scientific discovery agents, improving their ability to generate accurate and complete experiments.

Contribution

It introduces a system that distills large collections of scientific GitHub repositories into reliable code libraries for ASD agents, reducing manual effort and expanding capabilities.

Findings

74% of repositories yielded functional code examples

Augmented ASD agents produced more accurate and complete experiments

Moderate agreement between LLM-as-a-judge and domain-expert ratings

Abstract

Automated Scientific Discovery (ASD) systems can help automatically generate and run code-based experiments, but their capabilities are limited by the code they can reliably generate from parametric knowledge alone. As a result, current systems either mutate a small number of manually-crafted experiment examples, or operate solely from parametric knowledge, limiting quality and reach. We introduce CodeDistiller, a system that automatically distills large collections of scientific Github repositories into a vetted library of working domain-specific code examples, allowing ASD agents to expand their capabilities without manual effort. Using a combination of automatic and domain-expert evaluation on 250 materials science repositories, we find the best model is capable of producing functional examples for 74% of repositories, while our downstream evaluation shows an ASD agent augmented with a CodeDistiller generated library produces more accurate, complete, and scientifically sound experiments than an agent with only general materials-science code examples. We also evaluate LLM-as-a-judge ratings against domain-expert ratings in an A/B testing paradigm, finding moderate agreement and suggesting that inexpensive proxy metrics may be feasible for evaluating scientific discovery systems at scale.