Beyond Bias Scores: Unmasking Vacuous Neutrality in Small Language Models

TL;DR

This paper introduces VaNeu, a comprehensive multi-dimensional framework for evaluating fairness and reliability in small language models, revealing hidden vulnerabilities overlooked by traditional bias scores.

Contribution

The paper presents the first large-scale audit of 0.5-5B parameter SLMs using the novel VaNeu framework, highlighting limitations of existing bias assessments.

Findings

Models with low bias scores often fail in later evaluation stages.

Hidden vulnerabilities and unreliable reasoning are common in SLMs.

The framework aids responsible deployment in socially sensitive contexts.

Abstract

The rapid adoption of Small Language Models (SLMs) for resource constrained applications has outpaced our understanding of their ethical and fairness implications. To address this gap, we introduce the Vacuous Neutrality Framework (VaNeu), a multi-dimensional evaluation paradigm designed to assess SLM fairness prior to deployment. The framework examines model robustness across four stages - biases, utility, ambiguity handling, and positional bias over diverse social bias categories. To the best of our knowledge, this work presents the first large-scale audit of SLMs in the 0.5-5B parameter range, an overlooked "middle tier" between BERT-class encoders and flagship LLMs. We evaluate nine widely used SLMs spanning four model families under both ambiguous and disambiguated contexts. Our findings show that models demonstrating low bias in early stages often fail subsequent evaluations, revealing hidden vulnerabilities and unreliable reasoning. These results underscore the need for a more comprehensive understanding of fairness and reliability in SLMs, and position the proposed framework as a principled tool for responsible deployment in socially sensitive settings.