LIFT: Reinforcement Learning in Computer Systems by Learning From Demonstrations

TL;DR

LIFT is a deep reinforcement learning framework that leverages human demonstrations to efficiently optimize data management tasks in systems, outperforming traditional methods in key performance metrics.

Contribution

This work introduces LIFT, an end-to-end RL system utilizing demonstrations to reduce training time and a TensorFlow library, TensorForce, for practical deployment.

Findings

LIFT controllers outperform human baselines and heuristics by up to 70%.

Demonstration-based learning significantly reduces training time.

LIFT effectively improves latency and space usage in database and stream processing tasks.

Abstract

Reinforcement learning approaches have long appealed to the data management community due to their ability to learn to control dynamic behavior from raw system performance. Recent successes in combining deep neural networks with reinforcement learning have sparked significant new interest in this domain. However, practical solutions remain elusive due to large training data requirements, algorithmic instability, and lack of standard tools. In this work, we introduce LIFT, an end-to-end software stack for applying deep reinforcement learning to data management tasks. While prior work has frequently explored applications in simulations, LIFT centers on utilizing human expertise to learn from demonstrations, thus lowering online training times. We further introduce TensorForce, a TensorFlow library for applied deep reinforcement learning exposing a unified declarative interface to common RL algorithms, thus providing a backend to LIFT. We demonstrate the utility of LIFT in two case studies in database compound indexing and resource management in stream processing. Results show LIFT controllers initialized from demonstrations can outperform human baselines and heuristics across latency metrics and space usage by up to 70%.