Deep Learning and Handheld Augmented Reality Based System for Optimal Data Collection in Fault Diagnostics Domain

TL;DR

This paper introduces a novel human-machine interaction framework combining deep learning, reinforcement learning, and augmented reality to efficiently collect minimal data for fault diagnostics, achieving high accuracy with limited data.

Contribution

It presents a new integrated framework that leverages AR and reinforcement learning to optimize data collection for fault diagnostics, reducing data requirements and improving model performance.

Findings

Achieved over 100% precision and recall on a novel fault dataset.

The framework enables effective fault diagnosis with only one instance per fault.

All users successfully followed AR-guided data collection steps in usability tests.

Abstract

Compared to current AI or robotic systems, humans navigate their environment with ease, making tasks such as data collection trivial. However, humans find it harder to model complex relationships hidden in the data. AI systems, especially deep learning (DL) algorithms, impressively capture those complex relationships. Symbiotically coupling humans and computational machines' strengths can simultaneously minimize the collected data required and build complex input-to-output mapping models. This paper enables this coupling by presenting a novel human-machine interaction framework to perform fault diagnostics with minimal data. Collecting data for diagnosing faults for complex systems is difficult and time-consuming. Minimizing the required data will increase the practicability of data-driven models in diagnosing faults. The framework provides instructions to a human user to collect data that mitigates the difference between the data used to train and test the fault diagnostics model. The framework is composed of three components: (1) a reinforcement learning algorithm for data collection to develop a training dataset, (2) a deep learning algorithm for diagnosing faults, and (3) a handheld augmented reality application for data collection for testing data. The proposed framework has provided above 100\% precision and recall on a novel dataset with only one instance of each fault condition. Additionally, a usability study was conducted to gauge the user experience of the handheld augmented reality application, and all users were able to follow the provided steps.