Weakly Supervised Annotations for Multi-modal Greeting Cards Dataset

TL;DR

This paper introduces the Greeting Cards Dataset (GCD), a multimodal dataset with abstract concepts, and proposes methods to learn and generate greeting card images using pre-trained models for improved multi-modal understanding.

Contribution

The paper presents a new multimodal dataset with abstract concepts and demonstrates feature aggregation and captioning techniques using pre-trained models for greeting card applications.

Findings

Aggregated features improve understanding of abstract concepts.

Pre-trained models effectively generate greeting card images.

GCD enables multi-modal learning with limited supervision.

Abstract

In recent years, there is a growing number of pre-trained models trained on a large corpus of data and yielding good performance on various tasks such as classifying multimodal datasets. These models have shown good performance on natural images but are not fully explored for scarce abstract concepts in images. In this work, we introduce an image/text-based dataset called Greeting Cards. Dataset (GCD) that has abstract visual concepts. In our work, we propose to aggregate features from pretrained images and text embeddings to learn abstract visual concepts from GCD. This allows us to learn the text-modified image features, which combine complementary and redundant information from the multi-modal data streams into a single, meaningful feature. Secondly, the captions for the GCD dataset are computed with the pretrained CLIP-based image captioning model. Finally, we also demonstrate that the proposed the dataset is also useful for generating greeting card images using pre-trained text-to-image generation model.