Deep Reinforcement Learning for Backup Strategies against Adversaries

TL;DR

This paper introduces a reinforcement learning approach to optimize backup strategies in cybersecurity, specifically against adversaries capable of corrupting data at different times, outperforming traditional schemes.

Contribution

It models backup defense as a hybrid Markov decision process and applies deep reinforcement learning to discover superior backup schemes in adversarial settings.

Findings

RL-based schemes outperform traditional round-robin backups.

Optimal strategies adapt to adversarial timing of data corruption.

Deep deterministic policy gradients effectively solve the hybrid action space.

Abstract

Many defensive measures in cyber security are still dominated by heuristics, catalogs of standard procedures, and best practices. Considering the case of data backup strategies, we aim towards mathematically modeling the underlying threat models and decision problems. By formulating backup strategies in the language of stochastic processes, we can translate the challenge of finding optimal defenses into a reinforcement learning problem. This enables us to train autonomous agents that learn to optimally support planning of defense processes. In particular, we tackle the problem of finding an optimal backup scheme in the following adversarial setting: Given $k$ backup devices, the goal is to defend against an attacker who can infect data at one time but chooses to destroy or encrypt it at a later time, potentially also corrupting multiple backups made in between. In this setting, the usual round-robin scheme, which always replaces the oldest backup, is no longer optimal with respect to avoidable exposure. Thus, to find a defense strategy, we model the problem as a hybrid discrete-continuous action space Markov decision process and subsequently solve it using deep deterministic policy gradients. We show that the proposed algorithm can find storage device update schemes which match or exceed existing schemes with respect to various exposure metrics.