Investigating Antigram Behaviour using Distributional Semantics

TL;DR

This paper explores the concept of antigrams, semantically opposite anagrams, using distributional semantics and a rule-based approach, revealing initial challenges and the need for further research.

Contribution

It introduces a rule-based method to detect antigrams using GloVe embeddings and evaluates its performance on a small dataset.

Findings

Achieved 39% accuracy in antigram detection

Identified the complexity of semantically opposite anagram detection

Highlighted the need for improved algorithms in this area

Abstract

The field of computational linguistics constantly presents new challenges and topics for research. Whether it be analyzing word usage changes over time or identifying relationships between pairs of seemingly unrelated words. To this point, we identify Anagrams and Antigrams as words possessing such unique properties. The presented work is an exploration into generating anagrams from a given word and determining whether there exists antigram (semantically opposite anagrams) relationships between the pairs of generated anagrams using GloVe embeddings. We propose a rudimentary, yet interpretable, rule-based algorithm for detecting antigrams. On a small dataset of just 12 antigrams, our approach yielded an accuracy of 39\% which shows that there is much work left to be done in this space.

Tables

Table 1. Table 1: Semantic Similarity Scores for Antigram Testing

Word	Anagram	Anagram-Pair	Similarity Score	System Antigram	True Antigram
Abet	Bate	(abet, bate)	-0.02	No	No
	Beat	(abet, beat)	-0.1	Yes	Yes
	Beta	(abet, beta	0.13	No	No
		(bate, beat)	0.19	No	No
		(bate, beta)	-0.1	Yes	No
		(beat, beta)	0.01	No	No
Fakir	Kafir	(fakir, kafir)	0.21	No	Yes
Indeed	Denied	(indeed, denied)	0.43	No	Yes
Interim	Termini	(interim, termini)	-0.08	Yes	Yes
	Minteri	(interim, mintier)	0	N/A	No
		(termini, mintier)	0	N/A	No
OK	KO	(ok, ko)	0.29	No	Yes
Rousing	Souring	(rousing, souring)	0.08	No	Yes
Ruthless	Hurtless	(ruthless, hurtless)	0	N/A	Yes
	Hustlers	(ruthless, hustlers)	0.22	No	No
		(hurtless, hustlers)	0	N/A	No
Sainted	Instead	(sainted, instead)	-0.29	Yes	No
	Stained	(sainted, stained)	0.20	No	No
	Detains	(sainted, detains)	-0.13	Yes	No
		(stained, detains)	0.08	No	Yes
		(stained, instead)	-0.16	Yes	No
		(instead, detains)	-0.06	Yes	No
Scolded	Coddles	(scolded, coddles)	0.19	No	Yes
Sheared	Headers	(sheared, headers)	0.24	No	No
	Adheres	(sheared, adheres)	-0.05	Yes	Yes
		(headers, adheres)	-0.04	No	No
Striking	Skirting	(striking, skirting)	0.14	No	Yes
Tip	Pit	(tip, pit)	0.27	No	Yes

Table 2. Table 2: Semantic Similarity Scores for Antonym Pairs

Antonym	Similarity Score
Up-Down	0.92
Large-Small	0.93
Top-Bottom	0.59
Happy-Sad	0.68
Heavy-Light	0.64

Full text

1 Introduction

An anagram can be defined as a kind of word play where all the characters in the word are rearranged, using each character exactly once in order to generate a new word (s) which may or may not share semantic relationships with the root word. ‘Live’ and ‘Vile’ are examples of anagrams. Sometimes it is also possible to create a number of words from the root word using anagramming so as to produce a phrase; such is with the case ‘Dormitory’ and ‘Dirty Room’. Anagrams are present in a multitude of domains ranging from literature (a famous example is seen in the novel The Da Vinci Code where the phrase O, Draconian devil! was an anagram of Leonardo Da Vinci) to cyber security (for solving certain kinds of cryptograms such as the transposition and permutation ciphers). Antigrams on the other hand, are a class of anagrams which share an antonymic relationship with their anagram partner. For instance, ‘medicate’ and ‘decimate’ are examples of antigrams. These words are much more interesting to study because instead of a simple word play used to generate new phrases (anagrams) which might have a connection to the original word, the task can be to find a new word from the root word with which it has a strong relationship (antonym). Such exploration, while simple, can be used to probe the reasoning abilities of Large Language Models for example, as a task in benchmarks such as BigBench Srivastava et al. (2023) which includes such games as the high_low_game (think of a number and ask the language model to guess it by responding higher/lower than the guess).

In light of Natural Language Processing (NLP) tasks, very little work has been performed on antonyms and even less work has been done on anagrams and antigrams. Most of the work done has been conducted from the viewpoint of psychology experiments where researchers try to understand cognitive processing in the human mind. Adams et al. (2011) presented a study showing how the number of syllables influenced the difficulty of solving an anagram for both skilled and unskilled problem solvers. A similar task was undertaken by Novick and Sherman (2008). Vincent et al. (2006) created software which enabled users to discover novel anagrams and classify existing anagrams on the basis of certain psycholinguistic variables. Anagrams also help in understanding how cognition is linked with age or personality changes Java (1992). Thalenberg (2016) conducted research in exploring antonyms and distinguishing their presence in vector space by using Word2Vec embeddings Mikolov et al. (2013). But as mentioned before, none of the above works truly examined anagram or antigram relationships among words from the standpoint of NLP application. Rather they were explorations in cognitive science.

Generating single word anagrams is a relatively trivial task, simply compute every possible permutation of the characters of the given word and eliminate those terms which are noisy i.e. not found in either a corpus or a dictionary. However, trying to automatically detect antigrams among the generated anagrams in a trickier task since semantic information is required before such analysis can be conducted. This idea forms the main motivation of our work. We wanted to explore how antigrams are related to each other in terms of semantic similarity using word embedding models viz. GloVe embeddings, or in other words, automatically determine pairs of antigrams from the anagrams of a given word. The results obtained were compared with GloVe similarity scores computed between well-known antonyms and it was found that pure antonyms have, on average, higher similarity to each other than pairs of antigrams.

The rest of the paper is organized as follows. Section 2 describes the methodology of our work. Section 3 provides the results and analysis of experimentation. Finally, the paper is concluded and future directions are described in section 4.

2 Proposed Methodology

The key question which our system aims to address is whether pairs of anagrams generated from a target word share an antigram relationship. To solve this question, we devise a simple algorithm (cf. algorithm 1) which classifies a pair of words (anagrams) as having or not having antigram relationship on the basis of a threshold value which is set empirically.

Our work makes use of the GloVe model Pennington et al. (2014) for calculating semantic similarity between the generated anagrams. GloVe, one of the earliest attempts at creating dense word representations, is an unsupervised learning model which learns real numbered vectors, by analyzing word co-occurrences in a given corpus. What differentiates GloVe from its predecessor viz. Word2Vec is the local v/s global context scope in learning embeddings i.e. while Word2Vec relies on generating pairs of positive and negative samples from the corpus, using a given window size (thus local context) GloVe learns its embeddings from a big word co-occurrence matrix it constructs over the entire corpus (thus global view) and optimizes for a word association probability objective function.

In the proposed algorithm, each permutation (P) of the root word (W) is run through a spellchecker which was implemented in our work with the help of the PyEnchant111https://pythonhosted.org/pyenchant/ package for Python. A spell-checking module was required so as to filter out the invalid lexical forms of W (as all the permutations would not be proper English words). The members in the filtered permutation list are the anagrams that were required. Using their respective GloVe embedding, semantic similarity between each unique pair (C) of anagrams from the filtered permutation list was computed. The members of pair C were termed as $C_{0}$ and $C_{1}$ . Finally, if the similarity score between $C_{0}$ and $C_{1}$ was found to be less than -0.06, the pair was selected as an antigram. We set the value to -0.06 for two reasons,

• •

Assuming antonym vectors point $180^{\circ}$ w.r.t each other, the cosine similarity in turn would be a negative score [$\cos(180^{\circ})=-1$].

• •

We arrived at 0.06 by taking an average of all the scores obtained during our trials.

A point of contention regarding the algorithm would be, the exclusive use of GloVe vectors for generating similarity scores between the anagram pairs. We provide three reasons for this choice,

• •

While contextualized embeddings from models such as BERT Devlin et al. (2019) and GPT-3 Brown et al. (2020) have been shown to outperform non-contextualized representations such as GloVe and Word2Vec, they typically tend to break down words to subwords during tokenization. As such, obtaining a unified representation for the whole word becomes non-trivial as there are a few choices to be considered, taking a mean of the subword embeddings, using the [CLS] token, etc.

• •

Additionally, given the very nature of contextualized embeddings i.e. learning representations of tokens on the fly given the surrounding context does not make it suitable when we want to analyze words and concepts in isolation i.e. how related or unrelated their vectors are with respect to other terms without the influence of guiding context.

• •

Finally, the main purpose of the algorithm was to examine the nature of word vectors to see whether antonym relationship was equivalent to word vectors being opposite in direction in semantic space. As such, we hypothesized that the choice of embedding technique should not matter that much if the core assumptions for vector behavior hold (cf. 3).

3 Results Analysis

We compute semantic similarity using pre-trained GloVe embeddings, trained on the Wikipedia and Gigaword222https://catalog.ldc.upenn.edu/LDC2003T05 corpus. We use the 100-dimension variant vectors.

For testing our algorithm, the antigram dataset compiled by Anil (2010) was taken which consists of 50 pairs of well-known antigrams. This was the only reliable source of antigrams we could obtain seeing as it is such a niche area of study. However, we could only use 12 out of these 50 since the rest were multi-word antigrams and we wanted to focus our attention on single words rather than phrases, for the reasons mentioned in sec. 2 and also because phrases as a whole would not be present in GloVe’s vocabulary. Table 1 presents the results of the antigram tests.

From the results of Table 1, it became clear that a similarity score lower than -0.06 cannot be the only criteria for an anagram pair to be declared an antigram. With an overall accuracy of 39%, considering all 28 cases, a total of 3 pairs, out of 12 (25%), were actual (true) antigrams, which is quite low in modern architecture standards. Additionally, our model declared that a pair of anagrams were not antigrams more times than they were (16 v/s 12). This could be attributed to the low similarity threshold we set. Furthermore, our approach could not identify the other antigrams for two reasons, i) either the word does not exist in the models’ vocabulary (those pairs that are marked by N/A) or, ii) the similarity between the words is greater than -0.06. However, despite the poor performance, perhaps the only saving grace for our approach lies in its simplicity which leads to a more interpretable method allowing us to pinpoint where the decision error comes from.

Similarity scores were computed between well-known antonym pairs and a striking observation was made. Table 2 shows the corresponding results. The average semantic similarity score for the antonym pairs (cf. Table 2) was 0.75. Such a high score is actually counterintuitive. This is because the hypothesis surrounding regarding word vectors is that if they are antonyms, they would point in opposite directions and as such the cosine of the angle between them would be negative. Such a notion is clearly challenged when the scores from the antigram tests are contrasted with the antonym tests. The scores from Table 2 highlight the fact that it is not necessary for a word pair to have a negative similarity score, in order to be antonyms and vice versa. GloVe tries to reduce the angle between vectors of similar words and as such they become clustered very near to each other in vector space. Smaller the angle, closer is its cosine to 1. Thus, having a negative similarity score is not indicative of antonyms. Rather it was found that antonyms are strongly related to each other and as such produce high similarity scores. This fact was challenged by the antigrams whose similarity scores were extremely low and even negative in some cases (cf. Table 1) but still had an antonymic relationship. This indicates that antigrams and antonyms behave differently in semantic space in spite of sharing a common link. Extending the above logic, it could also mean that words that are considered antigrams are typically not found to share similar contexts thus resulting in lower similarity scores.

4 Conclusion and Future Work

The presented work aims to analyse antigrams and anagrams from the standpoint of NLP instead of treating them as subjects falling in the realm of logology (recreational linguistics). We propose a simple technique for detecting antigrams from pairs of anagrams using cosine semantic similarity. Our work is perhaps one of the first to explore this topic using distributional semantics. However, as can be seen from the paper, further research remains to be done. Firstly, figuring out a way to use contextual representations instead of static ones, which in theory should provide better context for computing similarity. Second, a more sophisticated approach to this problem could be to frame it as a text classification task. We could then take advantage of much larger models such as LLaMA Touvron et al. (2023) which have demonstrated impressive zero/few-shot performance across a range of tasks. Ultimately, in this paper, we propose basic ideas in the direction of anagram and antigram research and hope that future developments are undertaken towards it with implications for probing the reasoning abilities of large language models.

Limitations

Of course, there are many limitations with our work. Considering this as a pilot study in exploring the antigram phenomena, we identify three initial drawbacks, i) We focus only on unigram or single-word anagrams for detecting antigrams since GloVe cannot readily handle phrasal terms (needs further processing as mentioned before such as additive/mean of constituent embeddings) which limited us from using the entire dataset, ii) In this work we only relied on non-contextualized vectors. While they worked for our purposes here, it would certainly be interesting to see how we can leverage newer contextualized models such as BERT and GPT, and iii) We concede that the size of the dataset used was extremely small to obtain a clear picture of antigram behavior. Owing to the fact that there exists very limited resources for this task, the next undertaking can be to develop a larger and more diverse dataset for the same.

Ethics Statement

Seeing as this work ultimately computes semantic similarity between pairs of words, we believe that the ethical questions attached to it is minimal, if at all any. However, we believe that since the choice of vectors is important here, and by extension the corpus on which they were trained, care needs to be taken to not use corpora capable of perpetuating social biases in turn yielding scores indicative of the true meaning of the words without the influence of impeding biases.