Towards Improved Research Methodologies for Industrial AI: A case study of false call reduction

TL;DR

This paper critically examines current AI research methodologies through a case study on false call reduction in industrial inspection, highlighting shortcomings and proposing improvements for more effective applied AI practices.

Contribution

It identifies seven common weaknesses in existing research methodologies and demonstrates their impact using a real-world industrial AI case study.

Findings

Current best practices may fail in industrial AI applications.

Requirement-aware metrics are essential for aligning with business goals.

Analyzing temporal dynamics improves experimental validity.

Abstract

Are current artificial intelligence (AI) research methodologies ready to create successful, productive, and profitable AI applications? This work presents a case study on an industrial AI use case called false call reduction for automated optical inspection to demonstrate the shortcomings of current best practices. We identify seven weaknesses prevalent in related peer-reviewed work and experimentally show their consequences. We show that the best-practice methodology would fail for this use case. We argue amongst others for the necessity of requirement-aware metrics to ensure achieving business objectives, clear definitions of success criteria, and a thorough analysis of temporal dynamics in experimental datasets. Our work encourages researchers to critically assess their methodologies for more successful applied AI research.