GenZ: Foundational models as latent variable generators within traditional statistical models

TL;DR

GenZ introduces a hybrid approach combining foundational models and statistical methods to discover interpretable semantic features that improve prediction accuracy in domain-specific tasks like real estate and movie recommendations.

Contribution

The paper presents a novel generalized EM algorithm that jointly optimizes semantic feature descriptors and statistical model parameters, leveraging foundational models as latent variable generators.

Findings

Achieves 12% median relative error in house price prediction, outperforming GPT-5 baseline.

Predicts movie embeddings with 0.59 cosine similarity, matching performance of 4000 user ratings.

Discovers dataset-specific features that diverge from general domain knowledge.

Abstract

We present GenZ, a hybrid model that bridges foundational models and statistical modeling through interpretable semantic features. While large language models possess broad domain knowledge, they often fail to capture dataset-specific patterns critical for prediction tasks. Our approach addresses this by discovering semantic feature descriptions through an iterative process that contrasts groups of items identified via statistical modeling errors, rather than relying solely on the foundational model's domain understanding. We formulate this as a generalized EM algorithm that jointly optimizes semantic feature descriptors and statistical model parameters. The method prompts a frozen foundational model to classify items based on discovered features, treating these judgments as noisy observations of latent binary features that predict real-valued targets through learned statistical relationships. We demonstrate the approach on two domains: house price prediction (hedonic regression) and cold-start collaborative filtering for movie recommendations. On house prices, our model achieves 12\% median relative error using discovered semantic features from multimodal listing data, substantially outperforming a GPT-5 baseline (38\% error) that relies on the LLM's general domain knowledge. For Netflix movie embeddings, our model predicts collaborative filtering representations with 0.59 cosine similarity purely from semantic descriptions -- matching the performance that would require approximately 4000 user ratings through traditional collaborative filtering. The discovered features reveal dataset-specific patterns (e.g., architectural details predicting local housing markets, franchise membership predicting user preferences) that diverge from the model's domain knowledge alone.