Multi-Perspective LSTM for Joint Visual Representation Learning

TL;DR

This paper introduces a new LSTM cell architecture designed for learning complex relationships in multi-perspective visual sequences, improving recognition accuracy in tasks like lip reading and face recognition.

Contribution

The paper proposes a novel recurrent joint learning strategy with additional gates and memories, enhancing visual representation learning from multiple perspectives.

Findings

Superior recognition accuracy over benchmarks

Effective learning of intra- and inter-perspective relationships

Reduced complexity compared to existing methods

Abstract

We present a novel LSTM cell architecture capable of learning both intra- and inter-perspective relationships available in visual sequences captured from multiple perspectives. Our architecture adopts a novel recurrent joint learning strategy that uses additional gates and memories at the cell level. We demonstrate that by using the proposed cell to create a network, more effective and richer visual representations are learned for recognition tasks. We validate the performance of our proposed architecture in the context of two multi-perspective visual recognition tasks namely lip reading and face recognition. Three relevant datasets are considered and the results are compared against fusion strategies, other existing multi-input LSTM architectures, and alternative recognition solutions. The experiments show the superior performance of our solution over the considered benchmarks, both in terms of recognition accuracy and complexity. We make our code publicly available at https://github.com/arsm/MPLSTM.