Supervision Levels Scale (SLS)

TL;DR

This paper introduces a three-dimensional scale to quantify and compare the level of supervision used in training models, encompassing pre-training, labels, and data, applicable across tasks and datasets.

Contribution

It proposes the Supervision Levels Scale (SLS), a novel framework for encoding supervision levels, enabling better comparison of methods based on data supervision used.

Findings

SLS can be applied to any task/dataset/challenge.

Applied to EPIC-KITCHENS-100 for leaderboard integration.

Facilitates comparison of methods by supervision level.

Abstract

We propose a three-dimensional discrete and incremental scale to encode a method's level of supervision - i.e. the data and labels used when training a model to achieve a given performance. We capture three aspects of supervision, that are known to give methods an advantage while requiring additional costs: pre-training, training labels and training data. The proposed three-dimensional scale can be included in result tables or leaderboards to handily compare methods not only by their performance, but also by the level of data supervision utilised by each method. The Supervision Levels Scale (SLS) is first presented generally fo any task/dataset/challenge. It is then applied to the EPIC-KITCHENS-100 dataset, to be used for the various leaderboards and challenges associated with this dataset.