Multi-domain performance analysis with scores tailored to user preferences
S\'ebastien Pi\'erard, Adrien Deli\`ege, and Marc Van Droogenbroeck

TL;DR
This paper introduces a probabilistic framework for analyzing multi-domain algorithm performance, emphasizing user-preference-based scoring and visual tools for classification tasks.
Contribution
It proposes a novel theoretical approach to performance aggregation across domains, incorporating user preferences into scoring and domain classification.
Findings
Identification of scores that preserve weighted mean performance
Definition of four domain types based on user preferences
Development of new visual analysis tools for classification
Abstract
The performance of algorithms, methods, and models tends to depend heavily on the distribution of cases on which they are applied, this distribution being specific to the applicative domain. After performing an evaluation in several domains, it is highly informative to compute a (weighted) mean performance and, as shown in this paper, to scrutinize what happens during this averaging. To achieve this goal, we adopt a probabilistic framework and consider a performance as a probability measure (e.g., a normalized confusion matrix for a classification task). It appears that the corresponding weighted mean is known to be the summarization, and that only some remarkable scores assign to the summarized performance a value equal to a weighted arithmetic mean of the values assigned to the domain-specific performances. These scores include the family of ranking scores, a continuum parameterized…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsRecommender Systems and Techniques · Imbalanced Data Classification Techniques · Information Retrieval and Search Behavior
