Enhancing Understandability and Transparency of Research Software: Tracing Research to Code

TL;DR

This paper introduces an LLM-based tool that links research ideas in papers to their corresponding code, improving software understandability and transparency in research workflows.

Contribution

It presents a novel automation method that creates trace mappings between research ideas and code, aiding onboarding and review processes.

Findings

The tool can generate useful mappings between research ideas and code.

Initial experiments show promising results in automating idea-to-code tracing.

Abstract

Modern research heavily relies on software. A significant challenge researchers face is understanding the complex software used in specific research fields. We target two scenarios in this context, namely long onboarding times for newcomers and conference reviewers evaluating replication packages. We hypothesize that both scenarios can be significantly improved when there is a clear link between the paper's ideas and the code that implements them. As a time- and staff-saving approach, we propose an LLM-based automation tool that takes in a paper and the software implementing the paper, and generates a trace mapping between research ideas and their locations in code. Initial experiments have shown that the tool can generate quite useful mappings.