Transfer: Cross Modality Knowledge Transfer using Adversarial Networks -- A Study on Gesture Recognition

TL;DR

This paper introduces TRANSFER, a framework for cross-modality gesture recognition that leverages adversarial networks and language-based gesture representations to transfer knowledge between different sensing technologies.

Contribution

The paper presents a novel language-based approach and a generic adversarial framework for effective knowledge transfer across diverse sensing modalities in gesture recognition.

Findings

Successful transfer from video to WiFi for gesture recognition

Effective knowledge transfer from video to accelerometer signals

Cross-modality transfer from accelerometer to WiFi signals

Abstract

Knowledge transfer across sensing technology is a novel concept that has been recently explored in many application domains, including gesture-based human computer interaction. The main aim is to gather semantic or data driven information from a source technology to classify / recognize instances of unseen classes in the target technology. The primary challenge is the significant difference in dimensionality and distribution of feature sets between the source and the target technologies. In this paper, we propose TRANSFER, a generic framework for knowledge transfer between a source and a target technology. TRANSFER uses a language-based representation of a hand gesture, which captures a temporal combination of concepts such as handshape, location, and movement that are semantically related to the meaning of a word. By utilizing a pre-specified syntactic structure and tokenizer, TRANSFER segments a hand gesture into tokens and identifies individual components using a token recognizer. The tokenizer in this language-based recognition system abstracts the low-level technology-specific characteristics to the machine interface, enabling the design of a discriminator that learns technology-invariant features essential for recognition of gestures in both source and target technologies. We demonstrate the usage of TRANSFER for three different scenarios: a) transferring knowledge across technology by learning gesture models from video and recognizing gestures using WiFi, b) transferring knowledge from video to accelerometer, and d) transferring knowledge from accelerometer to WiFi signals.