Improving LLM Reliability through Hybrid Abstention and Adaptive Detection

TL;DR

This paper presents a dynamic, context-aware abstention system for LLMs that improves safety, reduces latency, and maintains high utility by integrating multiple detectors in a hierarchical cascade.

Contribution

It introduces an adaptive abstention framework with a multi-detector cascade that dynamically balances safety and utility in LLM deployment.

Findings

Reduces false positives in sensitive domains

Achieves lower latency compared to static guardrails

Maintains high safety recall and precision

Abstract

Large Language Models (LLMs) deployed in production environments face a fundamental safety-utility trade-off either a strict filtering mechanisms prevent harmful outputs but often block benign queries or a relaxed controls risk unsafe content generation. Conventional guardrails based on static rules or fixed confidence thresholds are typically context-insensitive and computationally expensive, resulting in high latency and degraded user experience. To address these limitations, we introduce an adaptive abstention system that dynamically adjusts safety thresholds based on real-time contextual signals such as domain and user history. The proposed framework integrates a multi-dimensional detection architecture composed of five parallel detectors, combined through a hierarchical cascade mechanism to optimize both speed and precision. The cascade design reduces unnecessary computation by progressively filtering queries, achieving substantial latency improvements compared to non-cascaded models and external guardrail systems. Extensive evaluation on mixed and domain-specific workloads demonstrates significant reductions in false positives, particularly in sensitive domains such as medical advice and creative writing. The system maintains high safety precision and near-perfect recall under strict operating modes. Overall, our context-aware abstention framework effectively balances safety and utility while preserving performance, offering a scalable solution for reliable LLM deployment.