Benchmarking Federated Learning for Semantic Datasets: Federated Scene Graph Generation

TL;DR

This paper introduces a novel federated learning benchmark for complex semantic datasets, specifically scene graphs, allowing controlled semantic heterogeneity across clients to evaluate federated methods on multi-semantic vision tasks.

Contribution

It proposes a new benchmark framework for federated learning with controllable semantic heterogeneity, addressing a gap in existing simple classification task benchmarks.

Findings

Demonstrates the effectiveness of existing PSG methods in federated settings.

Shows that robust federated algorithms improve performance under data heterogeneity.

Provides a new benchmark for evaluating federated learning on multi-semantic vision tasks.

Abstract

Federated learning (FL) enables decentralized training while preserving data privacy, yet existing FL benchmarks address relatively simple classification tasks, where each sample is annotated with a one-hot label. However, little attention has been paid to demonstrating an FL benchmark that handles complicated semantics, where each sample encompasses diverse semantic information, such as relations between objects. Because the existing benchmarks are designed to distribute data in a narrow view of a single semantic, managing the complicated semantic heterogeneity across clients when formalizing FL benchmarks is non-trivial. In this paper, we propose a benchmark process to establish an FL benchmark with controllable semantic heterogeneity across clients: two key steps are (i) data clustering with semantics and (ii) data distributing via controllable semantic heterogeneity across clients. As a proof of concept, we construct a federated PSG benchmark, demonstrating the efficacy of the existing PSG methods in an FL setting with controllable semantic heterogeneity of scene graphs. We also present the effectiveness of our benchmark by applying robust federated learning algorithms to data heterogeneity to show increased performance. To our knowledge, this is the first benchmark framework that enables federated learning and its evaluation for multi-semantic vision tasks under the controlled semantic heterogeneity. Our code is available at https://github.com/Seung-B/FL-PSG.