Fuzzy Logic Guided Reward Function Variation: An Oracle for Testing Reinforcement Learning Programs

TL;DR

This paper introduces a fuzzy logic-based automated oracle for testing reinforcement learning programs, effectively addressing the oracle problem by analyzing behavioral compliance trends and outperforming human oracles in complex scenarios.

Contribution

It presents a novel fuzzy logic-guided approach to automate RL program testing, improving reliability and scalability in complex environments.

Findings

Fuzzy oracle outperforms human oracles in complex RL testing scenarios.

The approach effectively detects buggy RL programs based on compliance trend violations.

Automated testing improves efficiency and scalability over manual methods.

Abstract

Reinforcement Learning (RL) has gained significant attention across various domains. However, the increasing complexity of RL programs presents testing challenges, particularly the oracle problem: defining the correctness of the RL program. Conventional human oracles struggle to cope with the complexity, leading to inefficiencies and potential unreliability in RL testing. To alleviate this problem, we propose an automated oracle approach that leverages RL properties using fuzzy logic. Our oracle quantifies an agent's behavioral compliance with reward policies and analyzes its trend over training episodes. It labels an RL program as "Buggy" if the compliance trend violates expectations derived from RL characteristics. We evaluate our oracle on RL programs with varying complexities and compare it with human oracles. Results show that while human oracles perform well in simpler testing scenarios, our fuzzy oracle demonstrates superior performance in complex environments. The proposed approach shows promise in addressing the oracle problem for RL testing, particularly in complex cases where manual testing falls short. It offers a potential solution to improve the efficiency, reliability, and scalability of RL program testing. This research takes a step towards automated testing of RL programs and highlights the potential of fuzzy logic-based oracles in tackling the oracle problem.