Automated Inference of Social Anxiety From Behavior in Social Virtual Reality: Cross-Sectional Observational Study

TL;DR

This study shows that social anxiety behaviors, like avoiding eye contact and speaking softly, can be detected in virtual reality interactions, similar to real-life settings.

Contribution

The study is the first to comprehensively examine behavioral and physiological markers of social anxiety in avatar-based human-human interactions in social VR.

Findings

Social anxiety was linked to reduced eye gaze, quieter speech, and lower heart rate variability in VR interactions.

Patterns of social anxiety overlapped with those of psychopathology but were opposite to those of verticality.

VR setups can capture rich behavioral data naturally, enabling new ways to detect and address social stress.

Abstract

Social anxiety often manifests through behaviors such as reduced gaze to the eyes and lower speech volume. While these markers have been examined in face-to-face interactions, large-scale assessments remain challenging. Social virtual reality (VR) offers a promising alternative by enabling naturalistic interactions in which behavior can be captured at scale, but it remains unclear if people show naturalistic behavior in such artificial environments.

We examined whether behavioral and physiological markers associated with social anxiety in real-life interactions similarly emerge in dyadic social VR. We additionally examined whether these patterns overlap with patterns linked to the broader constructs of psychopathology and verticality.

In this cross-sectional study, 128 participants (105 females, 22 males, 1 diverse; age 18-51 years; mean age 22.60, SD 3.57 years), recruited from a university student population, engaged in a 30-minute avatar-mediated dyadic conversation in social VR while physically located in separate rooms. We assessed gaze toward the partner’s eyes, smiling, and speaking behavior by using the VR headsets’ built-in eye trackers, face trackers, and microphones, and assessed high-frequency heart rate variability (HF-HRV) by using an electrocardiogram.

Relationships between traits and behavioral and physiological measures were analyzed using linear mixed-effects models (α=.05). Social anxiety was linked to reduced gaze toward the partner’s eyes while speaking (β=–.20, 95% CI –0.35 to –0.04; t126=–2.51; P=.01), quieter speech (β=–.18, 95% CI –0.35 to –0.01; t126=–2.12; P=.04), and reduced HF-HRV (β=–.23, 95% CI –0.39 to –0.08; t119=–3.00; P=.003). These findings were not entirely specific to social anxiety, as Pearson correlations revealed similar patterns for social anxiety and psychopathology (r=0.94, 95% CI 0.75-0.99; t7=7.61; P<.001), whereas verticality was linked to an overall opposite pattern (social anxiety: r=–0.92, 95% CI –0.98 to –0.65; t7=–6.14; P<.001; psychopathology: r=–0.83, 95% CI –0.96 to –0.38; t7=–3.98; P=.005).

In dyadic interactions in social VR, social anxiety was associated with behavioral and physiological modulations similar to those observed in face-to-face interactions, indicating heightened social stress and submissiveness even in avatar-mediated communication. Patterns were similar for heightened psychopathology and reversed for verticality, indicating that these traits may lie on a shared social-behavioral spectrum. Extending previous research focused on face-to-face interactions or reactions toward artificial agents displayed in VR, this study is the first to provide a comprehensive account of the behavioral and physiological modulations associated with social anxiety in avatar-based human-human interactions. Since social VR setups allow researchers to assess a rich set of behavioral data as a byproduct of the setup’s core functionality, the technique opens novel possibilities to detect social stress, track therapeutic progress, or tailor interventions to individual behavior when interactions take place in VR.

Figures