Decomposed Trust: Privacy, Adversarial Robustness, Ethics, and Fairness in Low-Rank LLMs

TL;DR

This study investigates how low-rank factorization impacts trustworthiness aspects of large language models, including privacy, robustness, ethics, and fairness, revealing both benefits and drawbacks across different settings.

Contribution

First comprehensive analysis of low-rank compressed LLMs on trustworthiness, combining empirical evaluation with explainability insights across multiple trust dimensions.

Findings

Low-rank factorization preserves training data privacy but weakens PII protection during conversations.

Adversarial robustness is generally improved by compression.

Ethics decline in zero-shot prompting but recover somewhat in few-shot settings.

Abstract

Large language models (LLMs) have driven major advances across domains, yet their massive size hinders deployment in resource-constrained settings. Low-rank factorization addresses this challenge by compressing models to effectively reduce their computation and memory consumption while maintaining accuracy. While these compressed models boast benign performance and system-level advantages, their trustworthiness implications remain poorly understood. In this paper, we present the first comprehensive study of how low-rank factorization affects LLM trustworthiness across privacy, adversarial robustness, ethics, and fairness, complemented by an explainability-driven analysis of the internal mechanisms behind these trust-related changes. We evaluate multiple LLMs of different sizes and architectures compressed with various low-rank factorization algorithms, revealing key insights: (1) low-rank factorization preserves training data privacy but weakens the protection of personally identifiable information during conversations; (2) adversarial robustness is generally enhanced under compression; (3) ethics degrades in zero-shot prompting but partially recovers in few-shot prompting; (4) fairness declines under compression. Beyond compression, we investigate how model scale and fine-tuning affect trustworthiness. Additionally, to move beyond black-box analysis, we employ a gradient-based attribution to identify which layers of LLMs contribute most to adversarial robustness.