JavelinGuard: Low-Cost Transformer Architectures for LLM Security

TL;DR

JavelinGuard introduces a suite of low-cost, high-performance transformer architectures optimized for detecting malicious intent in LLM interactions, balancing accuracy, speed, and resource efficiency for practical deployment.

Contribution

The paper presents novel transformer-based models specifically designed for LLM security, with comprehensive benchmarking across diverse datasets and comparison to existing guardrail models.

Findings

Raudra offers the most robust performance among the architectures.

All models outperform existing open-source guardrails in cost-performance trade-offs.

Models achieve high accuracy with fewer than 400M parameters.

Abstract

We present JavelinGuard, a suite of low-cost, high-performance model architectures designed for detecting malicious intent in Large Language Model (LLM) interactions, optimized specifically for production deployment. Recent advances in transformer architectures, including compact BERT(Devlin et al. 2019) variants (e.g., ModernBERT (Warner et al. 2024)), allow us to build highly accurate classifiers with as few as approximately 400M parameters that achieve rapid inference speeds even on standard CPU hardware. We systematically explore five progressively sophisticated transformer-based architectures: Sharanga (baseline transformer classifier), Mahendra (enhanced attention-weighted pooling with deeper heads), Vaishnava and Ashwina (hybrid neural ensemble architectures), and Raudra (an advanced multi-task framework with specialized loss functions). Our models are rigorously benchmarked across nine diverse adversarial datasets, including popular sets like the NotInject series, BIPIA, Garak, ImprovedLLM, ToxicChat, WildGuard, and our newly introduced JavelinBench, specifically crafted to test generalization on challenging borderline and hard-negative cases. Additionally, we compare our architectures against leading open-source guardrail models as well as large decoder-only LLMs such as gpt-4o, demonstrating superior cost-performance trade-offs in terms of accuracy, and latency. Our findings reveal that while Raudra's multi-task design offers the most robust performance overall, each architecture presents unique trade-offs in speed, interpretability, and resource requirements, guiding practitioners in selecting the optimal balance of complexity and efficiency for real-world LLM security applications.