AnglE-optimized Text Embeddings
Xianming Li, Jing Li
TL;DR
This paper introduces AnglE, a novel angle-optimized text embedding model that addresses vanishing gradient issues caused by cosine saturation, improving semantic textual similarity tasks across various datasets and domains.
Contribution
AnglE is the first model to incorporate angle optimization in complex space to mitigate cosine saturation effects in text embeddings.
Findings
AnglE outperforms state-of-the-art STS models on multiple datasets.
Angle optimization effectively mitigates cosine saturation issues.
The model is effective in domain-specific and limited data scenarios.
Abstract
High-quality text embedding is pivotal in improving semantic textual similarity (STS) tasks, which are crucial components in Large Language Model (LLM) applications. However, a common challenge existing text embedding models face is the problem of vanishing gradients, primarily due to their reliance on the cosine function in the optimization objective, which has saturation zones. To address this issue, this paper proposes a novel angle-optimized text embedding model called AnglE. The core idea of AnglE is to introduce angle optimization in a complex space. This novel approach effectively mitigates the adverse effects of the saturation zone in the cosine function, which can impede gradient and hinder optimization processes. To set up a comprehensive STS evaluation, we experimented on existing short-text STS datasets and a newly collected long-text STS dataset from GitHub Issues.…
Peer Reviews
Decision·ICLR 2024 Conference Withdrawn Submission
* The paper addresses an important issue in optimizing the cosine similarity of learning text embeddings, and the proposed method is interesting and novel. * It introduces the GitHub Issues Similarity Dataset as a testbed for evaluating model performance on long-text STS tasks. * The proposed method achieves promising results on a wide range of STS tasks.
* Some technical details are not clearly explained. For example, while the angle objective optimizes the text representations in a complex space, it's unclear how these complex vectors are obtained as the representations from language models are real vectors. * The paper seems to have missed discussions with a few important related studies. For example, [1] addresses the gradient vanishing issue by incorporating cosine distance in learning text embeddings, [2] designs angular softmax objectives
1. The proposed method of calculating similarity looks novel to me. 2. The impact of the method has the potential to be significant in many fields.
1. According to the paper, the motivation for introducing a complex space is to deal with the vanishing gradient of cos. In this sense, it would be great if techniques like gradient clipping and gradient normalization could be compared. 2. The writing can be improved. E.g., section 3.4 is a bit confusing to me. See my questions below. 3. I am also worried about the empirical significance. In table 2, the proposed method only improves the performance marginally (<1%) compared to SimCSE-BERT. I
1. This paper proposed a novel angle-optimized target to enhance the learning ability of contrastive learning-based representation learning models, which tried to alleviate the problem of vanishing gradients. 2. This paper developed a novel long-text STS dataset to better evaluate the performance of representation learning models. 3. This paper also explored LLM-based supervised data generation and contrastive learning, which is very interesting.
1. First of all, the authors argued that gradient vanishing problem is caused by the saturation zones in cosine functions in the optimization target. However, as far as I know, the gradient vanishing problem is mainly due to the deep structure. The saturation zones can be used to prove the high similarity between sentences. Therefore, the motivation of this paper is not so convincing. More explanations are needed. 2. Second, the authors focused on contrastive learning target, which limits the a
1. This paper identifies an interesting research question, the gradiant vanishing problem appearing at the saturation zone of cosine function influences the quality of text embeddings. 2. The proposed solution of using angle difference for optimization is orginal and novel. 3. Experiments on semantic textual similarity task are sufficiently conducted.
Despite an appealing motivation and an interesting solution, I still have the following concerns: 1. From my point of view, the only technical contribution of this paper is to design how to evaluate angle difference. This contribution is indeed interesting, but is a bit superficial and insufficient for a long research paper of ICLR standard. I expect authors to propose more __insightful__ designs to better solve the gradient vanishing problem. 2. The explanation of why saturation zone in cosin
Code & Models
- 🤗mixedbread-ai/mxbai-embed-large-v1model· 2.8M dl· ♡ 7762.8M dl♡ 776
- 🤗SeanLee97/angle-llama-7b-nli-20231027model· 9 dl· ♡ 39 dl♡ 3
- 🤗SeanLee97/angle-llama-7b-nli-v2model· 88 dl· ♡ 1988 dl♡ 19
- 🤗SeanLee97/angle-bert-base-uncased-nli-en-v1model· 585 dl· ♡ 5585 dl♡ 5
- 🤗SeanLee97/angle-llama-7b-zhnli-v1model· ♡ 2♡ 2
- 🤗SeanLee97/angle-roberta-wwm-base-zhnli-v1model· 184 dl· ♡ 3184 dl♡ 3
- 🤗SeanLee97/angle-llama-13b-nlimodel· 8 dl· ♡ 78 dl♡ 7
- 🤗WhereIsAI/UAE-Large-V1model· 1.4M dl· ♡ 2371.4M dl♡ 237
- 🤗odunola/UAE-Large-VImodel· 36 dl36 dl
- 🤗karrar-alwaili/UAE-Large-V1model· 1 dl1 dl
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Natural Language Processing Techniques · Text and Document Classification Technologies