Measuring and Eliminating Refusals in Military Large Language Models

TL;DR

This paper introduces a benchmark dataset to measure refusal rates in military large language models and explores methods to reduce refusals, aiming to improve their utility in time-critical military scenarios.

Contribution

It presents the first dataset for assessing refusal rates in military LLMs and evaluates techniques to significantly reduce refusals while maintaining task accuracy.

Findings

Refusal rates as high as 98.2% in some models

Synthetic datasets correlate with the gold refusal dataset

Ablation increased answer rate by 66.5 points

Abstract

Military Large Language Models (LLMs) must provide accurate information to the warfighter in time-critical and dangerous situations. However, today's LLMs are imbued with safety behaviors that cause the LLM to refuse many legitimate queries in the military domain, particularly those related to violence, terrorism, or military technology. Our gold benchmark for assessing refusal rates, which was developed by veterans of the US Army and special forces, is to our knowledge the first dataset of its kind. We present results for refusal and deflection rates on 31 public models and 3 military models. We observe hard rejection rates as high as 98.2% and soft deflection rates ranging from 0% to 21.3%. We also present results on two additional synthetic datasets and show their correlations with the gold dataset. Finally, we perform abliteration using the Heretic library on a military-tuned gpt-oss-20b model, showing an absolute increase in answer rate of 66.5 points but an average relative decrease of 2% on other military tasks. In our concluding remarks, we argue for deeper specialization, including with mid-training and end-to-end post-training, to achieve zero refusals and maximum military task accuracy for closed military models.