A Split-Window Transformer for Multi-Model Sequence Spammer Detection using Multi-Model Variational Autoencoder

TL;DR

This paper proposes MS$^2$Dformer, a novel Transformer architecture with a split-window multi-head attention mechanism and a multi-modal variational autoencoder, effectively detecting sequence spammers in multi-modal data.

Contribution

It introduces a hierarchical split-window attention mechanism and a user behavior tokenization algorithm for multi-modal sequence spammer detection.

Findings

MS$^2$Dformer outperforms previous state-of-the-art methods.

The hierarchical split-window attention effectively handles ultra-long sequences.

The model demonstrates strong generalization as a backbone for spam detection.

Abstract

This paper introduces a new Transformer, called MS$^2$Dformer, that can be used as a generalized backbone for multi-modal sequence spammer detection. Spammer detection is a complex multi-modal task, thus the challenges of applying Transformer are two-fold. Firstly, complex multi-modal noisy information about users can interfere with feature mining. Secondly, the long sequence of users' historical behaviors also puts a huge GPU memory pressure on the attention computation. To solve these problems, we first design a user behavior Tokenization algorithm based on the multi-modal variational autoencoder (MVAE). Subsequently, a hierarchical split-window multi-head attention (SW/W-MHA) mechanism is proposed. The split-window strategy transforms the ultra-long sequences hierarchically into a combination of intra-window short-term and inter-window overall attention. Pre-trained on the public datasets, MS$^2$Dformer's performance far exceeds the previous state of the art. The experiments demonstrate MS$^2$Dformer's ability to act as a backbone.