iPay: Integrated Payment Action Recognition via Multimodal Networks and Adaptive Spatial Prior Learning

TL;DR

iPay is a multimodal framework combining RGB and skeleton data with adaptive spatial priors for robust payment action recognition in transit surveillance, outperforming prior methods and suitable for edge deployment.

Contribution

The paper introduces iPay, a novel multimodal architecture with a prior-driven discriminator and dual-attention fusion, advancing automated payment action recognition in noisy onboard environments.

Findings

iPay achieves 83.45% recognition accuracy on real onboard footage.

The framework outperforms existing methods in accuracy and efficiency.

Collected over 55 hours of surveillance data with 500+ payment clips.

Abstract

Automated transit payment analysis is vital for scalable fare auditing and passenger analytics, yet practice still relies on limited manual inspection. Prior vision- and skeleton-based methods remain brittle under noisy onboard surveillance and often depend on poorly generalizable handcrafted features. Building on the success of graph convolutional networks in human action recognition, we observe that skeleton features excel at modeling global spatiotemporal dependencies but tend to underemphasize the subtle local relative motions that distinguish payment actions. In contrast, RGB features preserve fine-grained spatial details yet often lack reliable temporal continuity in surveillance footage. To bridge both system-level deployment needs and model-level design challenges, we present iPay, an integrated payment action recognition framework for onboard transit surveillance system. iPay adopts a multimodal mixture-of-experts architecture with four tightly coupled streams: (1) an RGB expert stream emphasizing local evidence via region-focused computation; (2) a skeleton expert stream modeling articulated motion with a graph convolutional backbone; (3) a dual-attention fusion stream enabling skeleton-to-RGB temporal transfer and RGB-to-skeleton spatial enhancement; and (4) a prior-driven Spatial Difference Discriminator (SDD) that explicitly models hand-to-anchor relative motion to improve task-specific discriminability. We also collaborate with local transit agencies to collect over 55 hours of real onboard surveillance footage, yielding 500+ payment clips. Experiments show that iPay outperforms prior methods and achieves 83.45\% recognition accuracy with competitive computational efficiency, making it suitable for edge deployment. Code is available at https://github.com/ccoopq/iPay.