Curating Grounded Synthetic Data with Global Perspectives for Equitable AI

TL;DR

This paper presents a method for creating diverse, culturally rich synthetic datasets from multilingual news sources to improve AI model robustness and generalizability, especially in data-scarce scenarios.

Contribution

We introduce a novel synthetic data generation approach grounded in real-world diversity, enhancing AI training data with multilingual, multicultural content for better generalization.

Findings

Up to 7.3% performance improvement on NER benchmarks

Synthetic data effectively captures global linguistic and cultural diversity

Methodology applicable to various AI domains for data diversification

Abstract

The development of robust AI models relies heavily on the quality and variety of training data available. In fields where data scarcity is prevalent, synthetic data generation offers a vital solution. In this paper, we introduce a novel approach to creating synthetic datasets, grounded in real-world diversity and enriched through strategic diversification. We synthesize data using a comprehensive collection of news articles spanning 12 languages and originating from 125 countries, to ensure a breadth of linguistic and cultural representations. Through enforced topic diversification, translation, and summarization, the resulting dataset accurately mirrors real-world complexities and addresses the issue of underrepresentation in traditional datasets. This methodology, applied initially to Named Entity Recognition (NER), serves as a model for numerous AI disciplines where data diversification is critical for generalizability. Preliminary results demonstrate substantial improvements in performance on traditional NER benchmarks, by up to 7.3%, highlighting the effectiveness of our synthetic data in mimicking the rich, varied nuances of global data sources. This paper outlines the strategies employed for synthesizing diverse datasets and provides such a curated dataset for NER.