STELLAR-E: a Synthetic, Tailored, End-to-end LLM Application Rigorous Evaluator

TL;DR

STELLAR-E is an automated system that generates synthetic datasets for evaluating large language models across multiple domains and languages, reducing reliance on manual data collection.

Contribution

It introduces a fully automated, controllable synthetic data generation framework and an evaluation pipeline that improves scalability and domain adaptability for LLM assessment.

Findings

Synthetic datasets achieved +5.7% LLM-as-a-judge score difference compared to existing benchmarks.

The system enables high-quality, scalable, and domain-specific dataset creation with minimal human input.

It provides a faster, automated alternative to manual dataset collection for LLM evaluation.

Abstract

The increasing reliance on Large Language Models (LLMs) across diverse sectors highlights the need for robust domain-specific and language-specific evaluation datasets; however, the collection of such datasets is challenging due to privacy concerns, regulatory restrictions, and the time cost for manual creation. Existing automated benchmarking methods are often limited by relying on pre-existing data, poor scalability, single-domain focus, and lack of multilingual support. We present STELLAR-E - a fully automated system to generate high-quality synthetic datasets of custom size, using minimal human inputs without depending on existing datasets. The system is structured in two stages: (1) We modify the TGRT Self-Instruct framework to create a synthetic data engine that enables controllable, custom synthetic dataset generation, and (2) an evaluation pipeline incorporating statistical and LLM-based metrics to assess the applicability of the synthetic dataset for LLM-based application evaluations. The synthetic datasets reach an average difference of +5.7% in terms of LLM-as-a-judge scores against existing language-specific benchmarks, demonstrating comparable quality for comprehensive assessment of big and small LLMs. While real datasets remain slightly more challenging for LLMs especially for smaller models, this work establishes a scalable and domain-adaptable benchmarking framework that supports fair evaluation of LLM applications, offering a faster alternative to manual approaches and enabling high-efficiency automated quality assurance cycles.