Detection, Retrieval, and Explanation Unified: A Violence Detection System Based on Knowledge Graphs and GAT

TL;DR

This paper introduces an interpretable violence detection system called TIO that combines knowledge graphs and graph attention networks to detect, retrieve, and explain violent behavior in videos, improving transparency and functionality.

Contribution

The paper presents a novel unified system integrating knowledge graphs and GAT for violence detection, retrieval, and explanation, with resource-efficient methods and validated effectiveness on benchmark datasets.

Findings

Effective violence detection and explanation demonstrated on XD-Violence and UCF-Crime datasets.

The system reveals that increasing bystanders correlates with decreased violence.

Lightweight methods improve system efficiency and resource consumption.

Abstract

Recently, violence detection systems developed using unified multimodal models have achieved significant success and attracted widespread attention. However, most of these systems face two critical challenges: the lack of interpretability as black-box models and limited functionality, offering only classification or retrieval capabilities. To address these challenges, this paper proposes a novel interpretable violence detection system, termed the Three-in-One (TIO) System. The TIO system integrates knowledge graphs (KG) and graph attention networks (GAT) to provide three core functionalities: detection, retrieval, and explanation. Specifically, the system processes each video frame along with text descriptions generated by a large language model (LLM) for videos containing potential violent behavior. It employs ImageBind to generate high-dimensional embeddings for constructing a knowledge graph, uses GAT for reasoning, and applies lightweight time series modules to extract video embedding features. The final step connects a classifier and retriever for multi-functional outputs. The interpretability of KG enables the system to verify the reasoning process behind each output. Additionally, the paper introduces several lightweight methods to reduce the resource consumption of the TIO system and enhance its efficiency. Extensive experiments conducted on the XD-Violence and UCF-Crime datasets validate the effectiveness of the proposed system. A case study further reveals an intriguing phenomenon: as the number of bystanders increases, the occurrence of violent behavior tends to decrease.