The One Where They Brain-Tune for Social Cognition: Multi-Modal Brain-Tuning on Friends
Nico Policzer, Cameron Braunstein, Mariya Toneva

TL;DR
This study extends brain-tuning of multi-modal models to enhance social cognition by aligning with the Superior Temporal Sulcus during video watching, improving both brain activity prediction and sarcasm detection in sitcoms.
Contribution
It introduces multi-modal brain-tuning targeting social brain regions, demonstrating improved brain alignment and social cognition task performance.
Findings
Increased brain alignment to STS during tuning
Enhanced sarcasm detection accuracy in sitcom context
Demonstrated effectiveness of multi-modal brain-tuning
Abstract
Recent studies on audio models show brain-tuning - fine-tuning models to better predict corresponding fMRI activity - improves brain alignment and increases performance on downstream semantic and audio tasks. We extend this approach to a multimodal audio-video model to enhance social cognition, targeting the Superior Temporal Sulcus (STS), a key region for social processing, while subjects watch Friends. We find significant increases in brain alignment to the STS and an adjacent ROI, as well as improvements to a social cognition task related to the training data - sarcasm detection in sitcoms. In summary, our study extends brain-tuning to the multi-modal domain, demonstrating improvements to a downstream task after tuning to a relevant functional region.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNeuroscience and Music Perception · Multisensory perception and integration · Emotion and Mood Recognition
