Solving a Real-World Optimization Problem Using Proximal Policy Optimization with Curriculum Learning and Reward Engineering

TL;DR

This paper introduces a curriculum learning-enhanced proximal policy optimization method with reward engineering to effectively train an agent for complex, real-world waste sorting, balancing safety, efficiency, and resource use.

Contribution

It presents a novel five-stage curriculum learning approach combined with reward engineering to train RL agents for complex industrial tasks with delayed rewards and class imbalance.

Findings

Achieved near-zero safety violations during inference

Significantly improved waste sorting efficiency

Demonstrated robustness in complex, real-world environment

Abstract

We present a proximal policy optimization (PPO) agent trained through curriculum learning (CL) principles and meticulous reward engineering to optimize a real-world high-throughput waste sorting facility. Our work addresses the challenge of effectively balancing the competing objectives of operational safety, volume optimization, and minimizing resource usage. A vanilla agent trained from scratch on these multiple criteria fails to solve the problem due to its inherent complexities. This problem is particularly difficult due to the environment's extremely delayed rewards with long time horizons and class (or action) imbalance, with important actions being infrequent in the optimal policy. This forces the agent to anticipate long-term action consequences and prioritize rare but rewarding behaviours, creating a non-trivial reinforcement learning task. Our five-stage CL approach tackles these challenges by gradually increasing the complexity of the environmental dynamics during policy transfer while simultaneously refining the reward mechanism. This iterative and adaptable process enables the agent to learn a desired optimal policy. Results demonstrate that our approach significantly improves inference-time safety, achieving near-zero safety violations in addition to enhancing waste sorting plant efficiency.