NYCU-TWO at Memotion 3: Good Foundation, Good Teacher, then you have Good Meme Analysis

TL;DR

This paper introduces a multi-modal meme sentiment analysis framework using CLIP features, a cooperative teaching model, and a cascaded classifier, achieving top rankings in the Memotion 3.0 shared task.

Contribution

It proposes a novel meme sentiment analysis framework combining CLIP, cooperative teaching, and cascaded classification, advancing multi-modal meme understanding.

Findings

Achieved 2nd place in Task A and B, 4th in Task C.

Weighted F1-scores of 0.342, 0.784, and 0.535 for Tasks A, B, and C.

Demonstrated robustness and effectiveness of the proposed framework.

Abstract

This paper presents a robust solution to the Memotion 3.0 Shared Task. The goal of this task is to classify the emotion and the corresponding intensity expressed by memes, which are usually in the form of images with short captions on social media. Understanding the multi-modal features of the given memes will be the key to solving the task. In this work, we use CLIP to extract aligned image-text features and propose a novel meme sentiment analysis framework, consisting of a Cooperative Teaching Model (CTM) for Task A and a Cascaded Emotion Classifier (CEC) for Tasks B&C. CTM is based on the idea of knowledge distillation, and can better predict the sentiment of a given meme in Task A; CEC can leverage the emotion intensity suggestion from the prediction of Task C to classify the emotion more precisely in Task B. Experiments show that we achieved the 2nd place ranking for both Task A and Task B and the 4th place ranking for Task C, with weighted F1-scores of 0.342, 0.784, and 0.535 respectively. The results show the robustness and effectiveness of our framework. Our code is released at github.