Interactive Elicitation of Knowledge on Feature Relevance Improves Predictions in Small Data Sets

TL;DR

This paper introduces an interactive visualization method to elicit domain experts' tacit knowledge about feature relevance, significantly enhancing prediction accuracy in small datasets with many features.

Contribution

It presents a novel user model-guided approach for eliciting expert knowledge to improve machine learning predictions in small data scenarios.

Findings

User model significantly improves prior knowledge elicitation

Enhanced prediction accuracy in small datasets

Effective in predicting scientific document citation counts

Abstract

Providing accurate predictions is challenging for machine learning algorithms when the number of features is larger than the number of samples in the data. Prior knowledge can improve machine learning models by indicating relevant variables and parameter values. Yet, this prior knowledge is often tacit and only available from domain experts. We present a novel approach that uses interactive visualization to elicit the tacit prior knowledge and uses it to improve the accuracy of prediction models. The main component of our approach is a user model that models the domain expert's knowledge of the relevance of different features for a prediction task. In particular, based on the expert's earlier input, the user model guides the selection of the features on which to elicit user's knowledge next. The results of a controlled user study show that the user model significantly improves prior knowledge elicitation and prediction accuracy, when predicting the relative citation counts of scientific documents in a specific domain.