Feature Relevance Determination for Ordinal Regression in the Context of Feature Redundancies and Privileged Information

TL;DR

This paper introduces a method for feature relevance determination in linear ordinal regression, addressing feature redundancies and privileged information to improve interpretability and identify all relevant features.

Contribution

It extends feature relevance intervals to ordinal regression, enabling identification of all relevant features and their relevance types, even with redundant or privileged features.

Findings

Identifies all strongly and weakly relevant features in ordinal regression.

Provides bounds for feature relevance considering redundancies.

Extends relevance intervals to privileged information scenarios.

Abstract

Advances in machine learning technologies have led to increasingly powerful models in particular in the context of big data. Yet, many application scenarios demand for robustly interpretable models rather than optimum model accuracy; as an example, this is the case if potential biomarkers or causal factors should be discovered based on a set of given measurements. In this contribution, we focus on feature selection paradigms, which enable us to uncover relevant factors of a given regularity based on a sparse model. We focus on the important specific setting of linear ordinal regression, i.e.\ data have to be ranked into one of a finite number of ordered categories by a linear projection. Unlike previous work, we consider the case that features are potentially redundant, such that no unique minimum set of relevant features exists. We aim for an identification of all strongly and all weakly relevant features as well as their type of relevance (strong or weak); we achieve this goal by determining feature relevance bounds, which correspond to the minimum and maximum feature relevance, respectively, if searched over all equivalent models. In addition, we discuss how this setting enables us to substitute some of the features, e.g.\ due to their semantics, and how to extend the framework of feature relevance intervals to the setting of privileged information, i.e.\ potentially relevant information is available for training purposes only, but cannot be used for the prediction itself.