LLM-Redactor: An Empirical Evaluation of Eight Techniques for Privacy-Preserving LLM Requests

TL;DR

This paper systematically evaluates eight privacy-preserving techniques for LLM requests, finding that combining local inference, redaction, and rephrasing offers effective privacy with minimal leaks.

Contribution

It introduces an open-source framework comparing eight privacy techniques for LLMs, providing practical guidance and a decision rule for selecting methods based on threat models.

Findings

No single technique dominates in privacy protection.

Combining local inference, redaction, and rephrasing reduces leaks significantly.

The open-source toolkit enables benchmarking and decision-making for privacy in LLMs.

Abstract

Coding agents and LLM-powered applications routinely send potentially sensitive content to cloud LLM APIs where it may be logged, retained, used for training, or subpoenaed. Existing privacy tooling focuses on network-level encryption and organization-level DLP, neither of which addresses the content of prompts themselves. We present a systematic empirical evaluation of eight techniques for privacy-preserving LLM requests: (A) local-only inference, (B) redaction with placeholder restoration, (C) semantic rephrasing, (D) Trusted Execution Environment hosted inference, (E) split inference, (F) fully homomorphic encryption, (G) secret sharing via multi-party computation, and (H) differential-privacy noise. We implement all eight (or a tractable research-stage subset where deployment is not yet feasible) in an open-source shim compatible with MCP and any OpenAI-compatible API. We evaluate the four practical options (A, B, C, H) and their combinations across four workload classes using a ground-truth-labelled leak benchmark of 1,300 samples with 4,014 annotations. Our headline finding is that no single technique dominates: the combination A+B+C (route locally when possible, redact and rephrase the rest) achieves 0.6% combined leak on PII and 31.3% on proprietary code, with zero exact leaks on PII across 500 samples. We present a decision rule that selects the appropriate option(s) from a threat-model budget and workload characterisation. Code, benchmarks, and evaluation harness are released at https://github.com/jayluxferro/llm-redactor.