Usability Evaluation for Online Professional Search in the Dutch Archaeology Domain

TL;DR

This paper introduces AGNES, a full-text search system for Dutch archaeological grey literature, evaluated through user studies to improve archaeologists' access to extensive excavation reports and enhance research efficiency.

Contribution

The paper presents AGNES, the first dedicated IR system for archaeological grey literature, with a user-centered evaluation revealing archaeologists' information needs and preferences.

Findings

Archaeologists' information needs are recall-oriented.

Users prefer free-text queries over metadata filters.

Diverse user feedback helps improve the system.

Abstract

This paper presents AGNES, the first information retrieval system for archaeological grey literature, allowing full-text search of these long archaeological documents. This search system has a web interface that allows archaeology professionals and scholars to search through a collection of over 60,000 Dutch excavation reports, totalling 361 million words. We conducted a user study for the evaluation of AGNES's search interface, with a small but diverse user group. The evaluation was done by screen capturing and a think aloud protocol, combined with a user interface feedback questionnaire. The evaluation covered both controlled use (completion of a pre-defined task) as well as free use (completion of a freely chosen task). The free use allows us to study the information needs of archaeologists, as well as their interactions with the search system. We conclude that: (1) the information needs of archaeologists are typically recall-oriented, often requiring a list of items as answer; (2) the users prefer the use of free-text queries over metadata filters, confirming the value of a free-text search system; (3) the compilation of a diverse user group contributed to the collection of diverse issues as feedback for improving the system. We are currently refining AGNES's user interface and improving its precision for archaeological entities, so that AGNES will help archaeologists to answer their research questions more effectively and efficiently, leading to a more coherent narrative of the past.