Beyond Accuracy: Measuring Representation Capacity of Embeddings to Preserve Structural and Contextual Information

TL;DR

This paper introduces a comprehensive method to evaluate the capacity of embeddings to preserve structural and contextual information, combining extrinsic and neighborhood analysis with optimization for objective assessment.

Contribution

It proposes a novel, data-driven evaluation framework for measuring the representation capacity of embeddings, integrating multiple metrics and optimization techniques.

Findings

Effective assessment of embedding quality using combined metrics

Optimization improves selection of the best embedding configurations

Evaluation on biological datasets demonstrates the method's utility

Abstract

Effective representation of data is crucial in various machine learning tasks, as it captures the underlying structure and context of the data. Embeddings have emerged as a powerful technique for data representation, but evaluating their quality and capacity to preserve structural and contextual information remains a challenge. In this paper, we address this need by proposing a method to measure the \textit{representation capacity} of embeddings. The motivation behind this work stems from the importance of understanding the strengths and limitations of embeddings, enabling researchers and practitioners to make informed decisions in selecting appropriate embedding models for their specific applications. By combining extrinsic evaluation methods, such as classification and clustering, with t-SNE-based neighborhood analysis, such as neighborhood agreement and trustworthiness, we provide a comprehensive assessment of the representation capacity. Additionally, the use of optimization techniques (bayesian optimization) for weight optimization (for classification, clustering, neighborhood agreement, and trustworthiness) ensures an objective and data-driven approach in selecting the optimal combination of metrics. The proposed method not only contributes to advancing the field of embedding evaluation but also empowers researchers and practitioners with a quantitative measure to assess the effectiveness of embeddings in capturing structural and contextual information. For the evaluation, we use $3$ real-world biological sequence (proteins and nucleotide) datasets and performed representation capacity analysis of $4$ embedding methods from the literature, namely Spike2Vec, Spaced $k$-mers, PWM2Vec, and AutoEncoder.