A New Type of Foundation Model Based on Recordings of People's Emotions and Physiology

TL;DR

This paper proposes a novel first-person foundation model trained on recordings of individuals' environmental stimuli and physiological responses, aiming to create more personalized and accurate AI systems.

Contribution

It introduces a new data collection rig and concept of first-person foundation models based on actual emotional and physiological responses, expanding foundation model capabilities.

Findings

Developed a recording rig for capturing environmental and physiological data.

Proposed the concept of first-person foundation models for personalized AI.

Discussed potential applications like recommendation engines and personal assistants.

Abstract

Foundation models have had a big impact in recent years and billions of dollars are being invested in them in the current AI boom. The more popular ones, such as Chat-GPT, are trained on large amounts of data from the Internet, and then reinforcement learning, RAG, prompt engineering and cognitive modelling are used to fine-tune and augment their behavior. This technology has been used to create models of individual people, such as Caryn Marjorie. However, these chatbots are not based on people's actual emotional and physiological responses to their environment, so they are, at best, surface-level approximations to the characters they are imitating. This paper describes how a new type of foundation model - a first-person foundation model - could be created from recordings of what a person sees and hears as well as their emotional and physiological reactions to these stimuli. A first-person foundation model would map environmental stimuli to a person's emotional and physiological states, and map a person's emotional and physiological states to their behavior. First-person foundation models have many exciting applications, including a new type of recommendation engine, personal assistants, generative adversarial networks, dating and recruitment. To obtain training data for a first-person foundation model, we have developed a recording rig that captures what the wearer is seeing and hearing as well as their emotional and physiological states. This novel source of data could help to address the shortage of new data for building the next generation of foundation models.