Modeling Object Attention in Mobile AR for Intrinsic Cognitive Security

TL;DR

This paper investigates how object recall in mobile AR is influenced by various factors, proposing a predictive model to enhance security by mitigating cognitive attacks that exploit recall vulnerabilities.

Contribution

It introduces a calibrated predictive model for object recall in mobile AR, combining SEM and machine learning to improve security and interface design.

Findings

PLS-SEM achieved the best F1 score in most studies.

Lighting, augmentation density, and AR stability are key recall drivers.

The model predicts recall probabilities to inform interface adjustments.

Abstract

We study attention in mobile Augmented Reality (AR) using object recall as a proxy outcome. We observe that the ability to recall an object (physical or virtual) that was encountered in a mobile AR experience depends on many possible impact factors and attributes, with some objects being readily recalled while others are not, and some people recalling objects overall much better or worse than others. This opens up a potential cognitive attack in which adversaries might create conditions that make an AR user not recall certain potentially mission-critical objects. We explore whether a calibrated predictor of object recall can help shield against such cognitive attacks. We pool data from four mobile AR studies (with a total of 1,152 object recall probes) and fit a Partial Least Squares Structural Equation Model (PLS-SEM) with formative Object, Scene, and User State composites predicting recall, also benchmarking against Random Forest and multilayer perceptron classifiers. PLS-SEM attains the best F1 score in three of four studies. Additionally, path estimates identify lighting, augmentation density, AR registration stability, cognitive load, and AR familiarity as primary drivers. The model outputs per-object recall probabilities that can drive interface adjustments when predicted recall falls. Overall, PLS-SEM provides competitive accuracy with interpretable levers for design and evaluation in mobile AR.