Logging Like Humans for LLMs: Rethinking Logging via Execution and Runtime Feedback

TL;DR

ReLog is an iterative framework that uses runtime feedback and LLMs to generate and refine logs, improving downstream debugging tasks over traditional static analysis methods.

Contribution

It introduces a runtime-guided, task-oriented logging generation approach that outperforms baselines by focusing on downstream utility rather than similarity to developer logs.

Findings

ReLog achieves an F1 score of 0.520 in defect localization.

ReLog repairs 97 defects in the direct debugging setting.

Iterative refinement and compilation repair are crucial for ReLog's effectiveness.

Abstract

Logging statements are essential for software debugging and maintenance. However, existing approaches to automatic logging generation rely on static analysis and produce statements in a single pass without considering runtime behavior. They are also typically evaluated by similarity to developer-written logs, assuming these logs form an adequate gold standard. This assumption is increasingly limiting in the LLM era, where logs are consumed not only by developers but also by LLMs for downstream tasks. As a result, optimizing logs for human similarity does not necessarily reflect their practical utility. To address these limitations, we introduce ReLog, an iterative logging generation framework guided by runtime feedback. ReLog leverages LLMs to generate, execute, evaluate, and refine logging statements so that runtime logs better support downstream tasks. Instead of comparing against developer-written logs, we evaluate ReLog through downstream debugging tasks, including defect localization and repair. We construct a benchmark based on Defects4J under both direct and indirect debugging settings. Results show that ReLog consistently outperforms all baselines, achieving an F1 score of 0.520 and repairing 97 defects in the direct setting, and the best F1 score of 0.408 in the indirect setting where source code is unavailable. Additional experiments across multiple LLMs demonstrate the generality of the framework, while ablations confirm the importance of iterative refinement and compilation repair. Overall, our work reframes logging as a runtime-guided, task-oriented process and advocates evaluating logs by their downstream utility rather than textual similarity.