SocialFusion: Addressing Social Degradation in Pre-trained Vision-Language Models

TL;DR

This paper identifies that current vision-language models suffer from social degradation due to pre-training, and proposes SocialFusion, a framework that improves social perception across multiple tasks by minimally connecting a frozen visual encoder with a language model.

Contribution

The paper introduces SocialFusion, a novel framework that mitigates social degradation in VLMs by minimally connecting a frozen visual encoder with a language model, enabling positive transfer across social tasks.

Findings

SocialFusion outperforms existing VLMs on five social perception tasks.

It achieves performance comparable to task-specific models on benchmarks.

Current pre-training strategies may hinder social competence acquisition.

Abstract

Understanding social interactions from visual cues is a fundamental challenge for a socially competent AI. While powerful pre-trained vision-language models (VLMs) have shown remarkable general capabilities, they surprisingly struggle to unify and learn multiple social perception tasks simultaneously, often exhibiting negative transfer. We identify that this negative transfer stems from a critical issue we term "social degradation," whereby the general visual-linguistic pre-training process of VLMs impairs the visual encoder's ability to represent nuanced social information. We investigate this behavior further under two lenses: decodability through linear representation probing and compatibility through gradient conflict analysis, revealing that both play a role in the degradation, especially the former, which is significantly compromised in the VLM pre-training process. To address these issues, we propose SocialFusion, a unified framework that learns a minimal connection between a frozen visual encoder and a language model. Compared with existing VLMs, it exhibits positive transfer across all five social tasks, leveraging synergies between them to enhance overall performance and achieves comparable performance to task-specific state-of-the-art models on various benchmarks. Our findings suggest that current VLM pre-training strategies may be detrimental to acquiring general social competence and highlight the need for more socially-aware training paradigms.