To Redact, or not to Redact? A Local LLM Approach to Deliberative Process Privilege Classification

TL;DR

This paper develops a local, small-scale LLM-based method for classifying sensitive government documents under FOIA exemptions, avoiding cloud API use and improving classification performance.

Contribution

It introduces a novel approach using a local Qwen3.5 9B model with advanced prompting techniques for sensitivity classification, outperforming previous models.

Findings

Chain-of-Thought and few-shot prompting improve recall and F2 score.

The local model approaches the performance of commercial models.

Sentences classified as deliberative contain more opinion-expressing verbs and first-person phrasing.

Abstract

Government transparency laws, like the Freedom of Information (FOIA) acts in the United States and United Kingdom, and the Woo (Open Government Act) in the Netherlands, grant citizens the right to directly request documents from the government. As these documents might contain sensitive information, such as personal information or threats to national security, the laws allow governments to redact sensitive parts of the documents prior to release. We build on prior research to perform automatic sensitivity classification for the FOIA Exemption 5 deliberative process privilege using Large Language Models (LLMs). However, processing documents not yet cleared for review via third-party cloud APIs is often legally or politically untenable. Therefore, in this work, we perform sensitivity classification with a small, local model, deployable on consumer-grade hardware (Qwen3.5 9B). We compare eight variants of applying LLMs for sentence classification, using well-known prompting techniques, and find that a combination of Chain-of-Thought prompting and few-shot prompting with error-based examples outperforms classification models of earlier work in terms of recall and F2 score. This method also closely approaches the performance of a widely-used, cost-efficient commercial model (Gemini 2.5 Flash). In an additional analysis, we find that sentences that are predicted as deliberative contain more verbs that indicate the expression of opinions, and are more often phrased in in first-person. Above all, deliberativeness seems characterized by the presence of a combination of multiple indicators, in particular the combination of first-person words with a verb for expressing opinion.