NodeSynth: Socially Aligned Synthetic Data for AI Evaluation

TL;DR

NodeSynth is a novel method for generating socially relevant synthetic data to evaluate AI models, revealing significant failure modes and deficiencies in safety guard models, with open-source tools for scalable assessment.

Contribution

We introduce NodeSynth, a new evidence-grounded approach for creating socially nuanced synthetic queries, enhancing AI evaluation in sensitive domains.

Findings

NodeSynth elicited failure rates up to five times higher than human benchmarks.

Granular taxonomic expansion significantly impacts failure rates.

Validation shows deficiencies in prominent guard models like Llama-Guard-3.

Abstract

Recent advancements in generative AI facilitate large-scale synthetic data generation for model evaluation. However, without targeted approaches, these datasets often lack the sociotechnical nuance required for sensitive domains. We introduce NodeSynth, an evidence-grounded methodology that generates socially relevant synthetic queries by leveraging a fine-tuned taxonomy generator (TaG) anchored in real-world evidence. Evaluated against four mainstream LLMs (e.g., Claude 4.5 Haiku), NodeSynth elicited failure rates up to five times higher than human-authored benchmarks. Ablation studies confirm that our granular taxonomic expansion significantly drives these failure rates, while independent validation reveals critical deficiencies in prominent guard models (e.g., Llama-Guard-3). We open-source our end-to-end research prototype and datasets to enable scalable, high-stakes model evaluation and targeted safety interventions (https://github.com/google-research/nodesynth).