Is Evaluating Visual Search Interfaces in Digital Libraries Still an Issue?

TL;DR

This paper discusses the ongoing challenge of evaluating visual search interfaces in digital libraries, emphasizing the need for standardized evaluation methods to improve comparability of results.

Contribution

It proposes establishing a common evaluation framework with standardized corpus, tasks, and reference system to enhance comparability of visual interface assessments.

Findings

Current evaluations lack standardization and comparability

A unified evaluation setup could improve assessment consistency

Standardized evaluation methods are needed for better interface comparison

Abstract

Although various visual interfaces for digital libraries have been developed in prototypical systems, very few of these visual approaches have been integrated into today's digital libraries. In this position paper we argue that this is most likely due to the fact that the evaluation results of most visual systems lack comparability. There is no fix standard on how to evaluate visual interactive user interfaces. Therefore it is not possible to identify which approach is more suitable for a certain context. We feel that the comparability of evaluation results could be improved by building a common evaluation setup consisting of a reference system, based on a standardized corpus with fixed tasks and a panel for possible participants.