Hybrid Affinity Propagation

TL;DR

This paper introduces H2MP, a hybrid affinity propagation method that efficiently summarizes tagged images by leveraging both visual and textual data, providing semantically meaningful visual and textual summaries.

Contribution

The paper proposes a novel scalar message propagation approach, H2MP, extending affinity propagation to handle heterogeneous visual and textual relations in image summarization.

Findings

H2MP effectively exploits visual and tag information for image summarization.

The approach produces semantically and visually satisfactory summaries.

Experimental results show improved efficiency and effectiveness over existing methods.

Abstract

In this paper, we address a problem of managing tagged images with hybrid summarization. We formulate this problem as finding a few image exemplars to represent the image set semantically and visually, and solve it in a hybrid way by exploiting both visual and textual information associated with images. We propose a novel approach, called homogeneous and heterogeneous message propagation ($\text{H}^\text{2}\text{MP}$). Similar to the affinity propagation (AP) approach, $\text{H}^\text{2}\text{MP}$ reduce the conventional \emph{vector} message propagation to \emph{scalar} message propagation to make the algorithm more efficient. Beyond AP that can only handle homogeneous data, $\text{H}^\text{2}\text{MP}$ generalizes it to exploit extra heterogeneous relations and the generalization is non-trivial as the reduction to scalar messages from vector messages is more challenging. The main advantages of our approach lie in 1) that $\text{H}^\text{2}\text{MP}$ exploits visual similarity and in addition the useful information from the associated tags, including the associations relation between images and tags and the relations within tags, and 2) that the summary is both visually and semantically satisfactory. In addition, our approach can also present a textual summary to a tagged image collection, which can be used to automatically generate a textual description. The experimental results demonstrate the effectiveness and efficiency of the roposed approach.