Out-Of-Distribution Detection for Audio-visual Generalized Zero-Shot Learning: A General Framework

TL;DR

This paper proposes a unified framework for audio-visual generalized zero-shot learning that integrates embedding and generative methods using out-of-distribution detection to improve classification of seen and unseen classes.

Contribution

It introduces a novel framework combining generative adversarial networks and OOD detection for audio-visual GZSL, addressing limitations of existing methods.

Findings

Significant performance improvements over state-of-the-art methods.

Effective synthesis of unseen features using GANs.

Robust OOD detection for better class differentiation.

Abstract

Generalized Zero-Shot Learning (GZSL) is a challenging task requiring accurate classification of both seen and unseen classes. Within this domain, Audio-visual GZSL emerges as an extremely exciting yet difficult task, given the inclusion of both visual and acoustic features as multi-modal inputs. Existing efforts in this field mostly utilize either embedding-based or generative-based methods. However, generative training is difficult and unstable, while embedding-based methods often encounter domain shift problem. Thus, we find it promising to integrate both methods into a unified framework to leverage their advantages while mitigating their respective disadvantages. Our study introduces a general framework employing out-of-distribution (OOD) detection, aiming to harness the strengths of both approaches. We first employ generative adversarial networks to synthesize unseen features, enabling the training of an OOD detector alongside classifiers for seen and unseen classes. This detector determines whether a test feature belongs to seen or unseen classes, followed by classification utilizing separate classifiers for each feature type. We test our framework on three popular audio-visual datasets and observe a significant improvement comparing to existing state-of-the-art works. Codes can be found in https://github.com/liuyuan-wen/AV-OOD-GZSL.