Towards Small Language Models for Security Query Generation in SOC Workflows

TL;DR

This paper explores the use of Small Language Models (SLMs) for translating natural language queries into Kusto Query Language (KQL) in security operations, aiming for accuracy and cost-efficiency.

Contribution

It introduces a three-knob framework involving prompting, fine-tuning, and architecture design to enhance SLM performance for security query generation.

Findings

Achieves high syntax (0.987) and semantic (0.906) accuracy on Microsoft's dataset.

Demonstrates up to 10x lower token cost compared to GPT-5.

Shows generalizability on Microsoft Sentinel data.

Abstract

Analysts in Security Operations Centers routinely query massive telemetry streams using Kusto Query Language (KQL). Writing correct KQL requires specialized expertise, and this dependency creates a bottleneck as security teams scale. This paper investigates whether Small Language Models (SLMs) can enable accurate, cost-effective natural-language-to-KQL translation for enterprise security. We propose a three-knob framework targeting prompting, fine-tuning, and architecture design. First, we adapt existing NL2KQL framework for SLMs with lightweight retrieval and introduce error-aware prompting that addresses common parser failures without increasing token count. Second, we apply LoRA fine-tuning with rationale distillation, augmenting each NLQ-KQL pair with a brief chain-of-thought explanation to transfer reasoning from a teacher model while keeping the SLM compact. Third, we propose a two-stage architecture that uses an SLM for candidate generation and a low-cost LLM judge for schema-aware refinement and selection. We evaluate nine models (five SLMs and four LLMs) across syntax correctness, semantic accuracy, table selection, and filter precision, alongside latency and token cost. On Microsoft's NL2KQL Defender Evaluation dataset, our two-stage approach achieves 0.987 syntax and 0.906 semantic accuracy. We further demonstrate generalizability on Microsoft Sentinel data, reaching 0.964 syntax and 0.831 semantic accuracy. These results come at up to 10x lower token cost than GPT-5, establishing SLMs as a practical, scalable foundation for natural-language querying in security operations.