LoFTI: Localization and Factuality Transfer to Indian Locales

TL;DR

This paper introduces LoFTI, a benchmark for evaluating how well large language models can transfer factual knowledge to Indian locales, revealing biases and localization capabilities in models like GPT-4.

Contribution

The paper presents LoFTI, a novel benchmark for assessing localization and factual transfer in LLMs, and evaluates multiple models on this new dataset.

Findings

Models show bias and skewed results in localized factual accuracy.

LoFTI effectively measures localization and factual transfer capabilities.

GPT-4 and others exhibit varying performance across hyperlocal levels.

Abstract

Large language models (LLMs) encode vast amounts of world knowledge acquired via training on large web-scale datasets crawled from the internet. However, these datasets typically exhibit a geographical bias towards English-speaking Western countries. This results in LLMs producing biased or hallucinated responses to queries that require answers localized to other geographical regions. In this work, we introduce a new benchmark named LoFTI (Localization and Factuality Transfer to Indian Locales) that can be used to evaluate an LLM's localization and factual text transfer capabilities. LoFTI consists of factual statements about entities in source and target locations; the source locations are spread across the globe and the target locations are all within India with varying degrees of hyperlocality (country, states, cities). The entities span a wide variety of categories. We use LoFTI to evaluate Mixtral, GPT-4 and two other Mixtral-based approaches well-suited to the task of localized factual transfer. We demonstrate that LoFTI is a high-quality evaluation benchmark and all the models, including GPT-4, produce skewed results across varying levels of hyperlocality.