Linnaeus: A highly reusable and adaptable ML based log classification pipeline

TL;DR

Linnaeus is an adaptable, reusable machine learning pipeline for log classification that addresses practical challenges like scalability, integration, and limited labeled data in large-scale software systems.

Contribution

The paper introduces Linnaeus, an end-to-end log classification pipeline emphasizing practicality, reusability, adaptability, and integration in industrial environments.

Findings

Demonstrates effective log classification with minimal labeled data

Shows high reusability and adaptability of the pipeline

Integrates ML solutions into large-scale software processes

Abstract

Logs are a common way to record detailed run-time information in software. As modern software systems evolve in scale and complexity, logs have become indispensable to understanding the internal states of the system. At the same time however, manually inspecting logs has become impractical. In recent times, there has been more emphasis on statistical and machine learning (ML) based methods for analyzing logs. While the results have shown promise, most of the literature focuses on algorithms and state-of-the-art (SOTA), while largely ignoring the practical aspects. In this paper we demonstrate our end-to-end log classification pipeline, Linnaeus. Besides showing the more traditional ML flow, we also demonstrate our solutions for adaptability and re-use, integration towards large scale software development processes, and how we cope with lack of labelled data. We hope Linnaeus can serve as a blueprint for, and inspire the integration of, various ML based solutions in other large scale industrial settings.