Self-supervised Hypergraphs for Learning Multiple World Interpretations

TL;DR

This paper introduces a self-supervised hypergraph approach for learning multiple scene representations and improving pretrained vision transformers using minimal labeled data, demonstrating superior performance on a new UAV dataset.

Contribution

The paper proposes a novel multi-task hypergraph framework for scene interpretation and enhances pretrained models without additional labels, introducing the Dronescapes dataset.

Findings

Hypergraph-based multi-task learning outperforms existing models.

Ensemble methods improve robustness and pseudolabel quality.

The approach effectively leverages minimal labeled data for complex scene understanding.

Abstract

We present a method for learning multiple scene representations given a small labeled set, by exploiting the relationships between such representations in the form of a multi-task hypergraph. We also show how we can use the hypergraph to improve a powerful pretrained VisTransformer model without any additional labeled data. In our hypergraph, each node is an interpretation layer (e.g., depth or segmentation) of the scene. Within each hyperedge, one or several input nodes predict the layer at the output node. Thus, each node could be an input node in some hyperedges and an output node in others. In this way, multiple paths can reach the same node, to form ensembles from which we obtain robust pseudolabels, which allow self-supervised learning in the hypergraph. We test different ensemble models and different types of hyperedges and show superior performance to other multi-task graph models in the field. We also introduce Dronescapes, a large video dataset captured with UAVs in different complex real-world scenes, with multiple representations, suitable for multi-task learning.