Global AI Bias Audit for Technical Governance

TL;DR

This paper conducts a global bias audit of LLMs, revealing significant geographic and socioeconomic disparities in AI knowledge, which threaten inclusive governance and global AI safety.

Contribution

It introduces a framework for stress-testing LLMs across diverse regions, exposing systemic biases and information gaps in current AI models.

Findings

AI knowledge is concentrated in high-income regions.

Lower-income countries face systemic information gaps.

Model responses are fact-based in only 11.4% of queries.

Abstract

This paper presents the outputs of the exploratory phase of a global audit of Large Language Models (LLMs) project. In this exploratory phase, I used the Global AI Dataset (GAID) Project as a framework to stress-test the Llama-3 8B model and evaluate geographic and socioeconomic biases in technical AI governance awareness. By stress-testing the model with 1,704 queries across 213 countries and eight technical metrics, I identified a significant digital barrier and gap separating the Global North and South. The results indicate that the model was only able to provide number/fact responses in 11.4% of its query answers, where the empirical validity of such responses was yet to be verified. The findings reveal that AI's technical knowledge is heavily concentrated in higher-income regions, while lower-income countries from the Global South are subject to disproportionate systemic information gaps. This disparity between the Global North and South poses concerning risks for global AI safety and inclusive governance, as policymakers in underserved regions may lack reliable data-driven insights or be misled by hallucinated facts. This paper concludes that current AI alignment and training processes reinforce existing geoeconomic and geopolitical asymmetries, and urges the need for more inclusive data representation to ensure AI serves as a truly global resource.