NoSQL Database Tuning through Machine Learning

TL;DR

This paper presents a machine learning-based approach to automatically optimize NoSQL database configurations, significantly improving throughput and reducing latency in Apache Cassandra through surrogate modeling and black-box optimization.

Contribution

It introduces a novel method using Random Forest and Gradient Boosting models to tune NoSQL databases, addressing the complexity of configuration inter-dependencies.

Findings

Up to 4% throughput improvement

Latency reductions of up to 43% (read) and 39% (write)

Feasibility demonstrated across various physical configurations

Abstract

NoSQL databases have become an important component of many big data and real-time web applications. Their distributed nature and scalability make them an ideal data storage repository for a variety of use cases. While NoSQL databases are delivered with a default ''off-the-shelf'' configuration, they offer configuration settings to adjust a database's behavior and performance to a specific use case and environment. The abundance and oftentimes imperceptible inter-dependencies of configuration settings make it difficult to optimize and performance-tune a NoSQL system. There is no one-size-fits-all configuration and therefore the workload, the physical design, and available resources need to be taken into account when optimizing the configuration of a NoSQL database. This work explores Machine Learning as a means to automatically tune a NoSQL database for optimal performance. Using Random Forest and Gradient Boosting Decision Tree Machine Learning algorithms, multiple Machine Learning models were fitted with a training dataset that incorporates properties of the NoSQL physical configuration (replication and sharding). The best models were then employed as surrogate models to optimize the Database Management System's configuration settings for throughput and latency using a Black-box Optimization algorithm. Using an Apache Cassandra database, multiple experiments were carried out to demonstrate the feasibility of this approach, even across varying physical configurations. The tuned DBMS configurations yielded throughput improvements of up to 4%, read latency reductions of up to 43%, and write latency reductions of up to 39% when compared to the default configuration settings.