Qualitative Judgement of Research Impact: Domain Taxonomy as a Fundamental Framework for Judgement of the Quality of Research

TL;DR

This paper proposes using domain taxonomy as a qualitative framework to assess research impact and quality, complementing traditional metrics through hierarchical analysis of research scope and influence.

Contribution

It introduces a novel approach of applying domain taxonomy for qualitative judgment of research impact, including a method for ranking researchers based on taxonomy modifications.

Findings

Taxonomies can analyze research scope and perspectives effectively.

Researcher ranking can be derived from taxonomy node transformations.

Experimental validation conducted in the data analysis domain.

Abstract

The appeal of metric evaluation of research impact has attracted considerable interest in recent times. Although the public at large and administrative bodies are much interested in the idea, scientists and other researchers are much more cautious, insisting that metrics are but an auxiliary instrument to the qualitative peer-based judgement. The goal of this article is to propose availing of such a well positioned construct as domain taxonomy as a tool for directly assessing the scope and quality of research. We first show how taxonomies can be used to analyse the scope and perspectives of a set of research projects or papers. Then we proceed to define a research team or researcher's rank by those nodes in the hierarchy that have been created or significantly transformed by the results of the researcher. An experimental test of the approach in the data analysis domain is described. Although the concept of taxonomy seems rather simplistic to describe all the richness of a research domain, its changes and use can be made transparent and subject to open discussions.