Recognition of Daily Activities through Multi-Modal Deep Learning: A Video, Pose, and Object-Aware Approach for Ambient Assisted Living

TL;DR

This paper introduces a multi-modal deep learning system combining video, pose, and object data to accurately recognize daily activities of older adults in indoor environments, enhancing Ambient Assisted Living systems.

Contribution

It presents a novel multi-modal approach integrating 3D CNN, GCN, and cross-attention mechanisms for activity recognition in AAL settings, evaluated on real-world datasets.

Findings

Achieves high classification accuracy on Toyota SmartHome dataset

Demonstrates robustness to environmental variability and scene complexity

Enhances activity recognition for older adults in indoor environments

Abstract

Recognition of daily activities is a critical element for effective Ambient Assisted Living (AAL) systems, particularly to monitor the well-being and support the independence of older adults in indoor environments. However, developing robust activity recognition systems faces significant challenges, including intra-class variability, inter-class similarity, environmental variability, camera perspectives, and scene complexity. This paper presents a multi-modal approach for the recognition of activities of daily living tailored for older adults within AAL settings. The proposed system integrates visual information processed by a 3D Convolutional Neural Network (CNN) with 3D human pose data analyzed by a Graph Convolutional Network. Contextual information, derived from an object detection module, is fused with the 3D CNN features using a cross-attention mechanism to enhance recognition accuracy. This method is evaluated using the Toyota SmartHome dataset, which consists of real-world indoor activities. The results indicate that the proposed system achieves competitive classification accuracy for a range of daily activities, highlighting its potential as an essential component for advanced AAL monitoring solutions. This advancement supports the broader goal of developing intelligent systems that promote safety and autonomy among older adults.