Reinforcement Learning Based Approaches to Adaptive Context Caching in Distributed Context Management Systems

TL;DR

This paper introduces reinforcement learning algorithms for adaptive context caching in distributed systems, significantly improving response times and reducing costs by efficiently reusing cached data without prior load knowledge.

Contribution

It presents novel RL-based algorithms with heuristic models for adaptive, selective context caching, including cache admission, eviction, and scaling strategies, outperforming traditional methods.

Findings

Achieves up to 60% cost efficiency improvement over traditional caching policies.

Demonstrates effectiveness of RL agents in adaptive context caching under synthetic loads.

Shows that selective caching strategies outperform cache-all policies.

Abstract

Performance metrics-driven context caching has a profound impact on throughput and response time in distributed context management systems for real-time context queries. This paper proposes a reinforcement learning based approach to adaptively cache context with the objective of minimizing the cost incurred by context management systems in responding to context queries. Our novel algorithms enable context queries and sub-queries to reuse and repurpose cached context in an efficient manner. This approach is distinctive to traditional data caching approaches by three main features. First, we make selective context cache admissions using no prior knowledge of the context, or the context query load. Secondly, we develop and incorporate innovative heuristic models to calculate expected performance of caching an item when making the decisions. Thirdly, our strategy defines a time-aware continuous cache action space. We present two reinforcement learning agents, a value function estimating actor-critic agent and a policy search agent using deep deterministic policy gradient method. The paper also proposes adaptive policies such as eviction and cache memory scaling to complement our objective. Our method is evaluated using a synthetically generated load of context sub-queries and a synthetic data set inspired from real world data and query samples. We further investigate optimal adaptive caching configurations under different settings. This paper presents, compares, and discusses our findings that the proposed selective caching methods reach short- and long-term cost- and performance-efficiency. The paper demonstrates that the proposed methods outperform other modes of context management such as redirector mode, and database mode, and cache all policy by up to 60% in cost efficiency.