LogDB: Multivariate Log-based Failure Diagnosis for Distributed Databases (Extended from MultiLog)

TL;DR

LogDB is a specialized log-based failure diagnosis system for distributed databases that extracts, compresses, and aggregates log features to effectively identify system anomalies, demonstrated on Apache IoTDB.

Contribution

This paper introduces LogDB, a novel failure diagnosis method tailored for distributed databases, leveraging log feature extraction and aggregation for improved anomaly detection.

Findings

Achieves robust failure diagnosis across various workloads.

Effectively detects different types of anomalies.

Demonstrated on Apache IoTDB with high accuracy.

Abstract

Distributed databases, as the core infrastructure software for internet applications, play a critical role in modern cloud services. However, existing distributed databases frequently experience system failures and performance degradation, often leading to significant economic losses. Log data, naturally generated within systems, can effectively reflect internal system states. In practice, operators often manually inspect logs to monitor system behavior and diagnose anomalies, a process that is labor-intensive and costly. Although various log-based failure diagnosis methods have been proposed, they are generally not tailored for database systems and fail to fully exploit the internal characteristics and distributed nature of these systems. To address this gap, we propose LogDB, a log-based failure diagnosis method specifically designed for distributed databases. LogDB extracts and compresses log features at each database node and then aggregates these features at the master node to diagnose cluster-wide anomalies. Experiments conducted on the open-source distributed database system Apache IoTDB demonstrate that LogDB achieves robust failure diagnosis performance across different workloads and a variety of anomaly types.