Team Triple-Check at Factify 2: Parameter-Efficient Large Foundation Models with Feature Representations for Multi-Modal Fact Verification

TL;DR

This paper introduces Pre-CoFactv2, a parameter-efficient multi-modal fact verification framework that outperforms previous models and achieves state-of-the-art results in the AAAI 2023 Factify challenge.

Contribution

The paper presents a novel parameter-efficient model with multi-modal fusion and feature representations, along with a unified ensemble method, advancing multi-modal fact verification.

Findings

Outperforms Pre-CoFact by a large margin

Achieves state-of-the-art F1-score of 81.82% at AAAI 2023

Team wins first prize in the Factify challenge

Abstract

Multi-modal fact verification has become an important but challenging issue on social media due to the mismatch between the text and images in the misinformation of news content, which has been addressed by considering cross-modalities to identify the veracity of the news in recent years. In this paper, we propose the Pre-CoFactv2 framework with new parameter-efficient foundation models for modeling fine-grained text and input embeddings with lightening parameters, multi-modal multi-type fusion for not only capturing relations for the same and different modalities but also for different types (i.e., claim and document), and feature representations for explicitly providing metadata for each sample. In addition, we introduce a unified ensemble method to boost model performance by adjusting the importance of each trained model with not only the weights but also the powers. Extensive experiments show that Pre-CoFactv2 outperforms Pre-CoFact by a large margin and achieved new state-of-the-art results at the Factify challenge at AAAI 2023. We further illustrate model variations to verify the relative contributions of different components. Our team won the first prize (F1-score: 81.82%) and we made our code publicly available at https://github.com/wwweiwei/Pre-CoFactv2-AAAI-2023.