Class-level Structural Relation Modelling and Smoothing for Visual Representation Learning

TL;DR

This paper introduces CSRMS, a framework that models class-level structural relations and applies smoothing to improve visual representation learning, especially for classes with diverse visual patterns, by leveraging relational graphs and graph convolution networks.

Contribution

The paper proposes a novel CSRMS framework that incorporates class-level relation modeling, graph sampling, and relational graph-guided learning to enhance visual representations.

Findings

CSRMS improves classification accuracy across multiple datasets.

Structured relation modeling enhances intra-class compactness.

The framework is compatible with various state-of-the-art models.

Abstract

Representation learning for images has been advanced by recent progress in more complex neural models such as the Vision Transformers and new learning theories such as the structural causal models. However, these models mainly rely on the classification loss to implicitly regularize the class-level data distributions, and they may face difficulties when handling classes with diverse visual patterns. We argue that the incorporation of the structural information between data samples may improve this situation. To achieve this goal, this paper presents a framework termed \textbf{C}lass-level Structural Relation Modeling and Smoothing for Visual Representation Learning (CSRMS), which includes the Class-level Relation Modelling, Class-aware Graph Sampling, and Relational Graph-Guided Representation Learning modules to model a relational graph of the entire dataset and perform class-aware smoothing and regularization operations to alleviate the issue of intra-class visual diversity and inter-class similarity. Specifically, the Class-level Relation Modelling module uses a clustering algorithm to learn the data distributions in the feature space and identify three types of class-level sample relations for the training set; Class-aware Graph Sampling module extends typical training batch construction process with three strategies to sample dataset-level sub-graphs; and Relational Graph-Guided Representation Learning module employs a graph convolution network with knowledge-guided smoothing operations to ease the projection from different visual patterns to the same class. Experiments demonstrate the effectiveness of structured knowledge modelling for enhanced representation learning and show that CSRMS can be incorporated with any state-of-the-art visual representation learning models for performance gains. The source codes and demos have been released at https://github.com/czt117/CSRMS.