Is my agent good enough? Evaluating Embodied Conversational Agents with Long and Short-term interactions

TL;DR

This paper presents a case study evaluating an Embodied Conversational Agent (ECA) using both long-term and short-term interactions to understand user perception and assess the agent's effectiveness over time.

Contribution

It introduces a methodology for evaluating ECAs through long-term interactions and compares it with short-term interactions to better understand user perception.

Findings

Long-term interactions provide unique insights into user perception.

Both LTI and STI are important for comprehensive ECA evaluation.

User perception varies across different interaction durations.

Abstract

The use of digital resources has been increasing in every instance of todays society, being it in business or even ludic purposes. Despite such ever increasing use of technologies as interfaces, in all fields, it seems that it lacks the importance of users perception in this context. This work aims to present a case study about the evaluation of ECAs. We propose a Long-Term Interaction (LTI) to evaluate our conversational agent effectiveness through the user perception and compare it with Short-Term Interactions (STIs), performed by three users. Results show that many different aspects of users perception about the chosen ECA (i.e. Arthur) could be evaluated in our case study, in particular that LTI and STI are both important in order to have a better understanding of ECA impact in UX.