Loghub: A Large Collection of System Log Datasets for AI-driven Log Analytics

TL;DR

Loghub is a comprehensive collection of 19 real-world system log datasets designed to advance AI-driven log analytics research and facilitate benchmarking in the field.

Contribution

This paper introduces loghub, a large, publicly available dataset collection for system logs, addressing the lack of open datasets and benchmarks in AI log analysis.

Findings

Loghub datasets have been downloaded approximately 90,000 times.

Benchmarking results demonstrate the utility of loghub for AI log analysis.

The datasets cover diverse system types, supporting broad research applications.

Abstract

Logs have been widely adopted in software system development and maintenance because of the rich runtime information they record. In recent years, the increase of software size and complexity leads to the rapid growth of the volume of logs. To handle these large volumes of logs efficiently and effectively, a line of research focuses on developing intelligent and automated log analysis techniques. However, only a few of these techniques have reached successful deployments in industry due to the lack of public log datasets and open benchmarking upon them. To fill this significant gap and facilitate more research on AI-driven log analytics, we have collected and released loghub, a large collection of system log datasets. In particular, loghub provides 19 real-world log datasets collected from a wide range of software systems, including distributed systems, supercomputers, operating systems, mobile systems, server applications, and standalone software. In this paper, we summarize the statistics of these datasets, introduce some practical usage scenarios of the loghub datasets, and present our benchmarking results on loghub to benefit the researchers and practitioners in this field. Up to the time of this paper writing, the loghub datasets have been downloaded for roughly 90,000 times in total by hundreds of organizations from both industry and academia. The loghub datasets are available at https://github.com/logpai/loghub.