Language Semantic Graph Guided Data-Efficient Learning

TL;DR

This paper introduces a Language Semantic Graph (LSG) that leverages label semantics via natural language descriptions to improve data-efficient learning across multiple modalities and scenarios, enhancing performance and training speed.

Contribution

The paper proposes a novel LSG approach that exploits label semantics with graph neural networks to guide model training, a new perspective in data-efficient learning.

Findings

LSG significantly improves performance in SSL and transfer learning.

LSG accelerates training processes across modalities.

The approach is versatile across image, video, and audio data.

Abstract

Developing generalizable models that can effectively learn from limited data and with minimal reliance on human supervision is a significant objective within the machine learning community, particularly in the era of deep neural networks. Therefore, to achieve data-efficient learning, researchers typically explore approaches that can leverage more related or unlabeled data without necessitating additional manual labeling efforts, such as Semi-Supervised Learning (SSL), Transfer Learning (TL), and Data Augmentation (DA). SSL leverages unlabeled data in the training process, while TL enables the transfer of expertise from related data distributions. DA broadens the dataset by synthesizing new data from existing examples. However, the significance of additional knowledge contained within labels has been largely overlooked in research. In this paper, we propose a novel perspective on data efficiency that involves exploiting the semantic information contained in the labels of the available data. Specifically, we introduce a Language Semantic Graph (LSG) which is constructed from labels manifest as natural language descriptions. Upon this graph, an auxiliary graph neural network is trained to extract high-level semantic relations and then used to guide the training of the primary model, enabling more adequate utilization of label knowledge. Across image, video, and audio modalities, we utilize the LSG method in both TL and SSL scenarios and illustrate its versatility in significantly enhancing performance compared to other data-efficient learning approaches. Additionally, our in-depth analysis shows that the LSG method also expedites the training process.