Chatting with Logs: An exploratory study on Finetuning LLMs for LogQL

TL;DR

This paper introduces NL2QL, a dataset and fine-tuning approach that significantly improves large language models' ability to generate accurate LogQL queries from natural language inputs, addressing challenges in log query standardization.

Contribution

The paper presents NL2QL, a new dataset and fine-tuning methodology that enhances LLMs' performance in log query generation, a previously underexplored area.

Findings

Fine-tuned models achieve up to 75% improvement in query accuracy.

NL2QL enables better training and evaluation of NL-to-LogQL systems.

Transferability across log formats is demonstrated through ablation studies.

Abstract

Logging is a critical function in modern distributed applications, but the lack of standardization in log query languages and formats creates significant challenges. Developers currently must write ad hoc queries in platform-specific languages, requiring expertise in both the query language and application-specific log details -- an impractical expectation given the variety of platforms and volume of logs and applications. While generating these queries with large language models (LLMs) seems intuitive, we show that current LLMs struggle with log-specific query generation due to the lack of exposure to domain-specific knowledge. We propose a novel natural language (NL) interface to address these inconsistencies and aide log query generation, enabling developers to create queries in a target log query language by providing NL inputs. We further introduce ~\textbf{NL2QL}, a manually annotated, real-world dataset of natural language questions paired with corresponding LogQL queries spread across three log formats, to promote the training and evaluation of NL-to-loq query systems. Using NL2QL, we subsequently fine-tune and evaluate several state of the art LLMs, and demonstrate their improved capability to generate accurate LogQL queries. We perform further ablation studies to demonstrate the effect of additional training data, and the transferability across different log formats. In our experiments, we find up to 75\% improvement of finetuned models to generate LogQL queries compared to non finetuned models.