Backprompting: Leveraging Synthetic Production Data for Health Advice Guardrails

TL;DR

This paper introduces backprompting, a method to generate synthetic, production-like labeled data for health advice guardrails, improving detector robustness with less data and outperforming larger models.

Contribution

The paper presents backprompting combined with human-in-the-loop clustering to create realistic training data for health advice detection in LLM outputs, enhancing guardrail effectiveness.

Findings

Backprompting generates realistic synthetic data for health advice detection.

The proposed detector outperforms GPT-4o by up to 3.73%.

Synthetic data improves guardrail robustness with significantly fewer parameters.

Abstract

The pervasiveness of large language models (LLMs) in enterprise settings has also brought forth a significant amount of risks associated with their usage. Guardrails technologies aim to mitigate this risk by filtering LLMs' input/output text through various detectors. However, developing and maintaining robust detectors faces many challenges, one of which is the difficulty in acquiring production-quality labeled data on real LLM outputs prior to deployment. In this work, we propose backprompting, a simple yet intuitive solution to generate production-like labeled data for health advice guardrails development. Furthermore, we pair our backprompting method with a sparse human-in-the-loop clustering technique to label the generated data. Our aim is to construct a parallel corpus roughly representative of the original dataset yet resembling real LLM output. We then infuse existing datasets with our synthetic examples to produce robust training data for our detector. We test our technique in one of the most difficult and nuanced guardrails: the identification of health advice in LLM output, and demonstrate improvement versus other solutions. Our detector is able to outperform GPT-4o by up to 3.73%, despite having 400x less parameters.