FEET: A Framework for Evaluating Embedding Techniques

TL;DR

FEET is a standardized evaluation framework for foundation models, covering three scenarios—frozen, few-shot, and fine-tuned—to comprehensively assess their practical performance across different applications.

Contribution

The paper introduces FEET, a structured protocol for benchmarking foundation models across multiple use cases, promoting consistency in evaluation methods.

Findings

Demonstrates FEET's applicability through sentiment analysis case study.

Shows FEET's effectiveness in medical domain model evaluation.

Provides a comprehensive assessment of foundation models' performance.

Abstract

In this study, we introduce FEET, a standardized protocol designed to guide the development and benchmarking of foundation models. While numerous benchmark datasets exist for evaluating these models, we propose a structured evaluation protocol across three distinct scenarios to gain a comprehensive understanding of their practical performance. We define three primary use cases: frozen embeddings, few-shot embeddings, and fully fine-tuned embeddings. Each scenario is detailed and illustrated through two case studies: one in sentiment analysis and another in the medical domain, demonstrating how these evaluations provide a thorough assessment of foundation models' effectiveness in research applications. We recommend this protocol as a standard for future research aimed at advancing representation learning models.