A Methodology to Evaluate Strategies Predicting Rankings on Unseen Domains

TL;DR

This paper introduces a methodology to predict the performance rankings of various strategies on unseen domains, reducing the need for costly new evaluations, demonstrated with background subtraction methods across multiple video domains.

Contribution

It proposes a leave-one-domain-out evaluation framework for predicting strategy rankings on unseen domains, applicable to diverse application-specific preferences.

Findings

Successfully predicts rankings of 40 entities on 53 domains

Reduces need for new costly evaluations in domain adaptation

Applicable to various strategies and application contexts

Abstract

Frequently, multiple entities (methods, algorithms, procedures, solutions, etc.) can be developed for a common task and applied across various domains that differ in the distribution of scenarios encountered. For example, in computer vision, the input data provided to image analysis methods depend on the type of sensor used, its location, and the scene content. However, a crucial difficulty remains: can we predict which entities will perform best in a new domain based on assessments on known domains, without having to carry out new and costly evaluations? This paper presents an original methodology to address this question, in a leave-one-domain-out fashion, for various application-specific preferences. We illustrate its use with 30 strategies to predict the rankings of 40 entities (unsupervised background subtraction methods) on 53 domains (videos).