Adobe Summit Concierge Evaluation with Human in the Loop

TL;DR

This paper presents Summit Concierge, an AI assistant for Adobe Summit, developed with a human-in-the-loop approach to handle real-world constraints and improve deployment scalability.

Contribution

It introduces a domain-specific AI assistant built with prompt engineering, retrieval grounding, and human validation, demonstrating effective deployment in enterprise events.

Findings

Scalable development through agile, feedback-driven processes

Effective handling of data sparsity and quality issues

Successful real-world deployment outcomes

Abstract

Generative AI assistants offer significant potential to enhance productivity, streamline information access, and improve user experience in enterprise contexts. In this work, we present Summit Concierge, a domain-specific AI assistant developed for Adobe Summit. The assistant handles a wide range of event-related queries and operates under real-world constraints such as data sparsity, quality assurance, and rapid deployment. To address these challenges, we adopt a human-in-the-loop development workflow that combines prompt engineering, retrieval grounding, and lightweight human validation. We describe the system architecture, development process, and real-world deployment outcomes. Our experience shows that agile, feedback-driven development enables scalable and reliable AI assistants, even in cold-start scenarios.