# TSLNet: a hierarchical multi-head attention-enabled two-stream LSTM network for accurate pedestrian tracking and behavior recognition

## TL;DR

TSLNet is a new deep learning model that improves pedestrian tracking and behavior recognition in complex environments using attention mechanisms and LSTM networks.

## Contribution

The novel contribution is the Hierarchical Multi-Head Attention-Enabled Two-Stream LSTM Network for joint tracking and behavior recognition.

## Key findings

- TSLNet outperforms existing models in behavior recognition metrics like Accuracy, Precision, and F1-Score.
- The model achieves superior performance in pedestrian tracking with higher MOTA and IDF1 scores.
- It effectively handles environmental variability and dense crowds using a multi-head attention mechanism.

## Abstract

Accurate pedestrian tracking and behavior recognition are essential for intelligent surveillance, smart transportation, and human-computer interaction systems. This paper introduces TSLNet, a Hierarchical Multi-Head Attention-Enabled Two-Stream LSTM Network, designed to overcome challenges such as environmental variability, high-density crowds, and diverse pedestrian movements in real-world video data. TSLNet combines a Two-Stream Convolutional Neural Network (Two-Stream CNN) with Long Short-Term Memory (LSTM) networks to effectively capture spatial and temporal features. The addition of a Multi-Head Attention mechanism allows the model to focus on relevant features in complex environments, while Hierarchical Classifiers within a Multi-Task Learning framework enable the simultaneous recognition of basic and complex behaviors. Experimental results on multiple public and proprietary datasets demonstrate that TSLNet significantly outperforms existing baseline models, achieving higher Accuracy, Precision, Recall, F1-Score, and Mean Average Precision (mAP) in behavior recognition, as well as superior Multiple Object Tracking Accuracy (MOTA) and ID F1 Score (IDF1) in pedestrian tracking. These improvements highlight TSLNet’s effectiveness in enhancing tracking and recognition performance.