Sycamore: Characterizing Synthetic Personas for Evaluating Genomics Visualization Retrieval

TL;DR

Sycamore investigates how grounded versus ungrounded synthetic personas influence the evaluation of genomics visualization tools, revealing their alignment and divergence from real expert feedback.

Contribution

This study introduces a three-condition probe design to assess the impact of grounding synthetic personas in genomics visualization evaluation.

Findings

Grounded synthetic personas align more with documented user concerns.

Ungrounded personas focus on operational details not raised by real users.

Both synthetic conditions miss certain image-modality preferences of real experts.

Abstract

Evaluating visualization systems in niche domains such as genomics is challenging due to scarcity of domain experts and difficulty recruiting a representative user base. While LLM-based synthetic personas are increasingly used to ease evaluation bottlenecks, they face well-founded skepticism. Rather than weighing synthetic personas as substitutes for real users, we ask a fundamental open question: when synthetic personas evaluate a real visualization system, what do they actually produce, and how does that output change when grounded in documented human contexts? We present Sycamore, an exploratory three-condition probe design using Geranium, a search engine for multimodal genomics visualization, as a case study. Sycamore evaluates Geranium using: (1) ungrounded synthetic personas from generic LLM priors; (2) grounded synthetic personas constrained by voice-of-customer artifacts from a prior interview study; and (3) a published baseline study of real domain experts. We observe that grounding shifts synthetic feedback toward the language and concerns of documented users, while ungrounded evaluators drift toward operational specifics that real participants did not raise; both synthetic conditions, however, converge on a find-and-adapt frame and miss the image-modality preference observed in the expert study. We discuss what these observations imply for where synthetic personas might fit alongside expert studies in domain-specific visualization evaluation. All supplemental materials are available at https://osf.io/kdfr3/.