The Difficulty of Novelty Detection in Open-World Physical Domains: An Application to Angry Birds

TL;DR

This paper introduces a physics-based approach to quantify the difficulty of novelty detection in open-world physical environments, validated through a user study in the game Angry Birds, aligning with human perception.

Contribution

It presents a novel qualitative physics-based method to measure novelty detection difficulty in open-world domains, specifically applied and validated in Angry Birds.

Findings

Detection difficulty correlates with human perception.

Method accurately predicts human detection challenges.

Validated through user study in a physics simulation game.

Abstract

Detecting and responding to novel situations in open-world environments is a key capability of human cognition and is a persistent problem for AI systems. In an open-world, novelties can appear in many different forms and may be easy or hard to detect. Therefore, to accurately evaluate the novelty detection capability of AI systems, it is necessary to investigate how difficult it may be to detect different types of novelty. In this paper, we propose a qualitative physics-based method to quantify the difficulty of novelty detection focusing on open-world physical domains. We apply our method in the popular physics simulation game Angry Birds, and conduct a user study across different novelties to validate our method. Results indicate that our calculated detection difficulties are in line with those of human users.