CodeDistiller: Automatically Generating Code Libraries for Scientific Coding Agents
Peter Jansen, Samiah Hassan, Pragnya Narasimha

TL;DR
CodeDistiller automatically creates vetted, domain-specific code libraries from GitHub repositories to enhance scientific discovery agents, improving their ability to generate accurate and complete experiments.
Contribution
It introduces a system that distills large collections of scientific GitHub repositories into reliable code libraries for ASD agents, reducing manual effort and expanding capabilities.
Findings
74% of repositories yielded functional code examples
Augmented ASD agents produced more accurate and complete experiments
Moderate agreement between LLM-as-a-judge and domain-expert ratings
Abstract
Automated Scientific Discovery (ASD) systems can help automatically generate and run code-based experiments, but their capabilities are limited by the code they can reliably generate from parametric knowledge alone. As a result, current systems either mutate a small number of manually-crafted experiment examples, or operate solely from parametric knowledge, limiting quality and reach. We introduce CodeDistiller, a system that automatically distills large collections of scientific Github repositories into a vetted library of working domain-specific code examples, allowing ASD agents to expand their capabilities without manual effort. Using a combination of automatic and domain-expert evaluation on 250 materials science repositories, we find the best model is capable of producing functional examples for 74% of repositories, while our downstream evaluation shows an ASD agent augmented with…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMachine Learning in Materials Science · Scientific Computing and Data Management · Software Engineering Research
