From Judgment to Interference: Early Stopping LLM Harmful Outputs via Streaming Content Monitoring

TL;DR

This paper introduces a streaming content monitor trained for early detection of harmful outputs in LLMs, reducing latency and improving safety by making timely judgments during generation.

Contribution

It presents a novel data and model approach supporting partial detection, including a new dataset and a streaming monitor that outperforms existing methods in early harmfulness detection.

Findings

Achieves 0.95+ macro F1 score with only 18% of tokens observed

Outperforms full detection in early harmfulness identification

Enhances safety alignment by serving as a pseudo-harmfulness annotator

Abstract

Though safety alignment has been applied to most large language models (LLMs), LLM service providers generally deploy a subsequent moderation as the external safety guardrail in real-world products. Existing moderators mainly practice a conventional full detection, which determines the harmfulness based on the complete LLM output, causing high service latency. Recent works pay more attention to partial detection where moderators oversee the generation midway and early stop the output if harmfulness is detected, but they directly apply moderators trained with the full detection paradigm to incomplete outputs, introducing a training-inference gap that lowers the performance. In this paper, we explore how to form a data-and-model solution that natively supports partial detection. For the data, we construct FineHarm, a dataset consisting of 29K prompt-response pairs with fine-grained annotations to provide reasonable supervision for token-level training. Then, we propose the streaming content monitor, which is trained with dual supervision of response- and token-level labels and can follow the output stream of LLM to make a timely judgment of harmfulness. Experiments show that SCM gains 0.95+ in macro F1 score that is comparable to full detection, by only seeing the first 18% of tokens in responses on average. Moreover, the SCM can serve as a pseudo-harmfulness annotator for improving safety alignment and lead to a higher harmlessness score than DPO.