Evaluating Agent Interactions Through Episodic Knowledge Graphs

TL;DR

This paper introduces a novel evaluation method for conversational agents using episodic Knowledge Graphs, capturing knowledge accumulation over time to provide deeper qualitative insights into agent behavior.

Contribution

The paper presents a new graph-based evaluation framework that interprets conversational signals to assess agent performance beyond traditional metrics.

Findings

Knowledge-Graph-based evaluation offers richer qualitative insights.

The method correlates well with existing evaluation metrics.

It effectively captures the evolution of agent knowledge during interactions.

Abstract

We present a new method based on episodic Knowledge Graphs (eKGs) for evaluating (multimodal) conversational agents in open domains. This graph is generated by interpreting raw signals during conversation and is able to capture the accumulation of knowledge over time. We apply structural and semantic analysis of the resulting graphs and translate the properties into qualitative measures. We compare these measures with existing automatic and manual evaluation metrics commonly used for conversational agents. Our results show that our Knowledge-Graph-based evaluation provides more qualitative insights into interaction and the agent's behavior.