Is this conference a top-tier? ConfAssist: An assistive conflict resolution framework for conference categorization

TL;DR

ConfAssist is a framework that uses conference stability features to assist experts in accurately classifying conferences as top-tier or non top-tier, reducing subjective disagreements.

Contribution

It introduces a novel stability-based approach for conference categorization and provides an assistive system validated by expert surveys achieving over 85% accuracy.

Findings

Top-tier conferences show less fluctuation in stability features.

The system achieves 85.18% classification accuracy.

Initial signals of popularity can be identified for new conferences.

Abstract

Classifying publication venues into top-tier or non top-tier is quite subjective and can be debatable at times. In this paper, we propose ConfAssist, a novel assisting framework for conference categorization that aims to address the limitations in the existing systems and portals for venue classification. We start with the hypothesis that top-tier conferences are much more stable than other conferences and the inherent dynamics of these groups differs to a very large extent. We identify various features related to the stability of conferences that might help us separate a top-tier conference from the rest of the lot. While there are many clear cases where expert agreement can be almost immediately achieved as to whether a conference is a top-tier or not, there are equally many cases that can result in a conflict even among the experts. ConfAssist tries to serve as an aid in such cases by increasing the confidence of the experts in their decision. An analysis of 110 conferences from 22 sub-fields of computer science clearly favors our hypothesis as the top-tier conferences are found to exhibit much less fluctuations in the stability related features than the non top-tier ones. We evaluate our hypothesis using systems based on conference categorization. For the evaluation, we conducted human judgment survey with 28 domain experts. The results are impressive with 85.18% classification accuracy. We also compare the dynamics of the newly started conferences with the older conferences to identify the initial signals of popularity. The system is applicable to any conference with atleast 5 years of publication history.