Label Embedding with Partial Heterogeneous Contexts

TL;DR

This paper introduces PHCLE, a novel framework for label embedding that effectively integrates heterogeneous and partially observed contexts, improving label relatedness modeling in image classification and interpretability.

Contribution

The paper proposes a general framework for partial heterogeneous context label embedding, overcoming partial context issues and integrating multiple contexts with a shared embedding space.

Findings

PHCLE achieves superior image classification performance.

Embeddings exhibit good interpretability in label similarity analysis.

Effective optimization algorithm for sparse matrix factorization.

Abstract

Label embedding plays an important role in many real-world applications. To enhance the label relatedness captured by the embeddings, multiple contexts can be adopted. However, these contexts are heterogeneous and often partially observed in practical tasks, imposing significant challenges to capture the overall relatedness among labels. In this paper, we propose a general Partial Heterogeneous Context Label Embedding (PHCLE) framework to address these challenges. Categorizing heterogeneous contexts into two groups, relational context and descriptive context, we design tailor-made matrix factorization formula to effectively exploit the label relatedness in each context. With a shared embedding principle across heterogeneous contexts, the label relatedness is selectively aligned in a shared space. Due to our elegant formulation, PHCLE overcomes the partial context problem and can nicely incorporate more contexts, which both cannot be tackled with existing multi-context label embedding methods. An effective alternative optimization algorithm is further derived to solve the sparse matrix factorization problem. Experimental results demonstrate that the label embeddings obtained with PHCLE achieve superb performance in image classification task and exhibit good interpretability in the downstream label similarity analysis and image understanding task.