Adobe Summit Concierge Evaluation with Human in the Loop
Yiru Chen, Sally Fang, Sai Sree Harsha, Dan Luo, Vaishnavi Muppala, Fei Wu, Shun Jiang, Kun Qian, Yunyao Li

TL;DR
This paper presents Summit Concierge, an AI assistant for Adobe Summit, developed with a human-in-the-loop approach to handle real-world constraints and improve deployment scalability.
Contribution
It introduces a domain-specific AI assistant built with prompt engineering, retrieval grounding, and human validation, demonstrating effective deployment in enterprise events.
Findings
Scalable development through agile, feedback-driven processes
Effective handling of data sparsity and quality issues
Successful real-world deployment outcomes
Abstract
Generative AI assistants offer significant potential to enhance productivity, streamline information access, and improve user experience in enterprise contexts. In this work, we present Summit Concierge, a domain-specific AI assistant developed for Adobe Summit. The assistant handles a wide range of event-related queries and operates under real-world constraints such as data sparsity, quality assurance, and rapid deployment. To address these challenges, we adopt a human-in-the-loop development workflow that combines prompt engineering, retrieval grounding, and lightweight human validation. We describe the system architecture, development process, and real-world deployment outcomes. Our experience shows that agile, feedback-driven development enables scalable and reliable AI assistants, even in cold-start scenarios.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsSpreadsheets and End-User Computing · AI in Service Interactions · Persona Design and Applications
