Enhancing Trustworthiness with Mixed Precision: Benchmarks, Opportunities, and Challenges

TL;DR

This paper investigates how quantization affects the trustworthiness of large language models, revealing challenges and proposing a mixed-precision ensemble method that improves trustworthiness metrics in high-stakes applications.

Contribution

It systematically analyzes the impact of quantization on trustworthiness metrics and introduces a novel mixed-precision ensemble voting approach to enhance trustworthiness.

Findings

Quantization impacts adversarial robustness, fairness, ethics, and out-of-distribution robustness.

The proposed ensemble method improves trustworthiness metrics by up to 5.8%.

Identifies instability across compression ratios and quantization methods.

Abstract

Large language models (LLMs) have shown promising performance across various tasks. However, their autoregressive decoding process poses significant challenges for efficient deployment on existing AI hardware. Quantization alleviates memory and compute pressure by compressing weights, activations, and KV caches to low precisions while preserving generation quality. However, existing quantization frameworks typically focus on perplexity or classification accuracy, often omitting critical trustworthiness metrics. This gap introduces risks when applying quantized LLMs to downstream high-stakes domains such as finance and healthcare. In this work, we systematically investigate the impact of quantization on four trustworthiness metrics (adversarial robustness, fairness, machine ethics, and out-of-distribution robustness) and identify the instability across compression ratios and quantization methods. Building on these observations, we develop a novel precision-ensemble voting approach that leverages predictions from mixed-precision variants of the same model and consistently improves performance by up to $5.8\%$ on trustworthiness metrics. Our results highlight the importance of considering trustworthiness when developing model compression techniques and point to research opportunities at the intersection of compression and trustworthiness for safety-critical applications.