Category-Based Deep CCA for Fine-Grained Venue Discovery from Multimodal Data

TL;DR

This paper introduces a novel deep learning model called Category-based Deep CCA for fine-grained venue discovery using multimodal data, enabling exact and group venue searches by correlating images and textual descriptions.

Contribution

The paper proposes a new deep CCA model that jointly optimizes cross-modal correlations for venue discovery, incorporating category-based grouping and a new multimodal dataset.

Findings

Outperforms state-of-the-art methods in cross-modal retrieval tasks.

Effectively captures venue features from multimodal data.

Enables accurate venue search using user-generated photos.

Abstract

In this work, travel destination and business location are taken as venues. Discovering a venue by a photo is very important for context-aware applications. Unfortunately, few efforts paid attention to complicated real images such as venue photos generated by users. Our goal is fine-grained venue discovery from heterogeneous social multimodal data. To this end, we propose a novel deep learning model, Category-based Deep Canonical Correlation Analysis (C-DCCA). Given a photo as input, this model performs (i) exact venue search (find the venue where the photo was taken), and (ii) group venue search (find relevant venues with the same category as that of the photo), by the cross-modal correlation between the input photo and textual description of venues. In this model, data in different modalities are projected to a same space via deep networks. Pairwise correlation (between different modal data from the same venue) for exact venue search and category-based correlation (between different modal data from different venues with the same category) for group venue search are jointly optimized. Because a photo cannot fully reflect rich text description of a venue, the number of photos per venue in the training phase is increased to capture more aspects of a venue. We build a new venue-aware multimodal dataset by integrating Wikipedia featured articles and Foursquare venue photos. Experimental results on this dataset confirm the feasibility of the proposed method. Moreover, the evaluation over another publicly available dataset confirms that the proposed method outperforms state-of-the-arts for cross-modal retrieval between image and text.