Relevance As a Metric for Evaluating Machine Learning Algorithms

TL;DR

This paper introduces Relevance Score, a new probability-based metric for evaluating supervised learning algorithms, demonstrating its effectiveness over accuracy in specific application contexts through empirical analysis.

Contribution

The paper proposes the Relevance Score as a novel evaluation metric that better reflects user concerns in certain applications compared to traditional accuracy.

Findings

Relevance Score outperforms accuracy in selected application scenarios.

Empirical analysis on lighting data validates the effectiveness of Relevance Score.

Relevance Score aligns better with user relevance in specific domains.

Abstract

In machine learning, the choice of a learning algorithm that is suitable for the application domain is critical. The performance metric used to compare different algorithms must also reflect the concerns of users in the application domain under consideration. In this work, we propose a novel probability-based performance metric called Relevance Score for evaluating supervised learning algorithms. We evaluate the proposed metric through empirical analysis on a dataset gathered from an intelligent lighting pilot installation. In comparison to the commonly used Classification Accuracy metric, the Relevance Score proves to be more appropriate for a certain class of applications.