psifx -- Psychological and Social Interactions Feature Extraction Package

TL;DR

psifx is a modular, open-source toolkit that automates multi-modal data annotation and feature extraction to support large-scale psychological and social science research.

Contribution

It introduces a comprehensive, community-driven software package that simplifies and standardizes multi-modal data processing for human sciences.

Findings

Supports tasks like speaker diarization, transcription, and pose estimation.

Enables large-scale, real-time behavioral analysis in social science research.

Facilitates non-expert access to advanced machine learning tools.

Abstract

psifx is a plug-and-play multi-modal feature extraction toolkit, aiming to facilitate and democratize the use of state-of-the-art machine learning techniques for human sciences research. It is motivated by a need (a) to automate and standardize data annotation processes that typically require expensive, lengthy, and inconsistent human labour; (b) to develop and distribute open-source community-driven psychology research software; and (c) to enable large-scale access and ease of use for non-expert users. The framework contains an array of tools for tasks such as speaker diarization, closed-caption transcription and translation from audio; body, hand, and facial pose estimation and gaze tracking with multi-person tracking from video; and interactive textual feature extraction supported by large language models. The package has been designed with a modular and task-oriented approach, enabling the community to add or update new tools easily. This combination creates new opportunities for in-depth study of real-time behavioral phenomena in psychological and social science research.