Reward Engineering for Reinforcement Learning in Software Tasks

TL;DR

This paper systematically reviews reward engineering techniques for reinforcement learning applied to software tasks, highlighting the diversity of reward signals and design choices in this rapidly growing field.

Contribution

It provides the first comprehensive survey of reward design approaches in RL for software, organizing existing methods along key dimensions and discussing future challenges.

Findings

Reward signals in software RL are often proxies like compilation or test pass rates.

Reward design is scattered across multiple areas with no unified overview.

The survey identifies key challenges and offers recommendations for future reward engineering.

Abstract

Reinforcement learning is increasingly used for code-centric tasks. These tasks include code generation, summarization, understanding, repair, testing, and optimization. This trend is growing faster with large language models and autonomous agents. A key challenge is how to design reward signals that make sense for software. In many RL problems, the reward is a clear number. In software, this is often not possible. The goal is rarely a single numeric objective. Instead, rewards are usually proxies. Common proxies check if the code compiles, passes tests, or satisfies quality metrics. Many reward designs have been proposed for code-related tasks. However, the work is scattered across areas and papers. There is no single survey that brings these approaches together and shows the full landscape of reward design for RL in software. In this survey, we provide the first systematic and comprehensive review of reward engineering for RL in software tasks. We focus on existing methods and techniques. We structure the literature along three complementary dimensions, summarizing the reward-design choices within each. We conclude with challenges and recommendations in the reward design space for SE tasks.