MAGNeto: An Efficient Deep Learning Method for the Extractive Tags Summarization Problem

TL;DR

MAGNeto introduces a deep learning approach for extractive image tag summarization that combines visual and textual data, auxiliary losses, gating, data augmentation, and unsupervised pre-training to improve performance on benchmark datasets.

Contribution

The paper presents a novel unified architecture integrating multiple deep learning components and strategies for extractive tags summarization, including a new loss function and pre-training method.

Findings

Achieves 90% F1 score on NUS-WIDE benchmark.

Attains 50% F1 score on a large-scale private dataset.

Demonstrates effectiveness of unsupervised pre-training and data augmentation.

Abstract

In this work, we study a new image annotation task named Extractive Tags Summarization (ETS). The goal is to extract important tags from the context lying in an image and its corresponding tags. We adjust some state-of-the-art deep learning models to utilize both visual and textual information. Our proposed solution consists of different widely used blocks like convolutional and self-attention layers, together with a novel idea of combining auxiliary loss functions and the gating mechanism to glue and elevate these fundamental components and form a unified architecture. Besides, we introduce a loss function that aims to reduce the imbalance of the training data and a simple but effective data augmentation technique dedicated to alleviates the effect of outliers on the final results. Last but not least, we explore an unsupervised pre-training strategy to further boost the performance of the model by making use of the abundant amount of available unlabeled data. Our model shows the good results as 90% $F_\text{1}$ score on the public NUS-WIDE benchmark, and 50% $F_\text{1}$ score on a noisy large-scale real-world private dataset. Source code for reproducing the experiments is publicly available at: https://github.com/pixta-dev/labteam