Harnessing Large Language Model for Virtual Reality Exploration Testing: A Case Study

TL;DR

This study explores using GPT-4o, a large language model, for VR exploration testing, demonstrating its ability to identify, describe, and compare entities in virtual environments with high accuracy, and discussing its limitations.

Contribution

The paper presents a case study validating LLMs for VR FOV analysis, showing prompt engineering improves entity identification and scene understanding capabilities.

Findings

Prompt engineering increases test entity identification accuracy from 41.67% to 71.30%.

LLMs can describe entity features with at least 90% accuracy.

Core features like color, placement, and shape improve entity matching in VR.

Abstract

As the Virtual Reality (VR) industry expands, the need for automated GUI testing is growing rapidly. Large Language Models (LLMs), capable of retaining information long-term and analyzing both visual and textual data, are emerging as a potential key to deciphering the complexities of VR's evolving user interfaces. In this paper, we conduct a case study to investigate the capability of using LLMs, particularly GPT-4o, for field of view (FOV) analysis in VR exploration testing. Specifically, we validate that LLMs can identify test entities in FOVs and that prompt engineering can effectively enhance the accuracy of test entity identification from 41.67% to 71.30%. Our study also shows that LLMs can accurately describe identified entities' features with at least a 90% accuracy rate. We further find out that the core features that effectively represent an entity are color, placement, and shape. Furthermore, the combination of the three features can especially be used to improve the accuracy of determining identical entities in multiple FOVs with the highest F1-score of 0.70. Additionally, our study demonstrates that LLMs are capable of scene recognition and spatial understanding in VR with precisely designed structured prompts. Finally, we find that LLMs fail to label the identified test entities, and we discuss potential solutions as future research directions.