InCA: Rethinking In-Car Conversational System Assessment Leveraging Large Language Models

TL;DR

This paper proposes new KPIs and datasets for evaluating in-car conversational question answering systems using large language models, addressing limitations of existing metrics and enhancing assessment through persona variation.

Contribution

It introduces tailored KPIs and datasets for in-car ConvQA evaluation and demonstrates the benefits of using varied personas in prompts.

Findings

Proposed KPIs effectively evaluate in-car ConvQA systems.

Datasets designed for specific KPIs improve assessment accuracy.

Persona variation in prompts enhances model evaluation diversity.

Abstract

The assessment of advanced generative large language models (LLMs) poses a significant challenge, given their heightened complexity in recent developments. Furthermore, evaluating the performance of LLM-based applications in various industries, as indicated by Key Performance Indicators (KPIs), is a complex undertaking. This task necessitates a profound understanding of industry use cases and the anticipated system behavior. Within the context of the automotive industry, existing evaluation metrics prove inadequate for assessing in-car conversational question answering (ConvQA) systems. The unique demands of these systems, where answers may relate to driver or car safety and are confined within the car domain, highlight the limitations of current metrics. To address these challenges, this paper introduces a set of KPIs tailored for evaluating the performance of in-car ConvQA systems, along with datasets specifically designed for these KPIs. A preliminary and comprehensive empirical evaluation substantiates the efficacy of our proposed approach. Furthermore, we investigate the impact of employing varied personas in prompts and found that it enhances the model's capacity to simulate diverse viewpoints in assessments, mirroring how individuals with different backgrounds perceive a topic.