What do Toothbrushes do in the Kitchen? How Transformers Think our World is Structured
Alexander Henlein, Alexander Mehler

TL;DR
This paper investigates how transformer-based language models encode object relations, comparing their knowledge extraction capabilities with static models using various similarity measures and classifiers.
Contribution
It provides a comparative analysis of contextualized versus static models in extracting object relation knowledge, highlighting the impact of similarity measures and classifiers.
Findings
Models with different similarity measures vary greatly in knowledge extraction.
Classifier-based approaches outperform similarity measures.
Static models perform nearly as well as, or better than, contextualized models.
Abstract
Transformer-based models are now predominant in NLP. They outperform approaches based on static models in many respects. This success has in turn prompted research that reveals a number of biases in the language models generated by transformers. In this paper we utilize this research on biases to investigate to what extent transformer-based language models allow for extracting knowledge about object relations (X occurs in Y; X consists of Z; action A involves using X). To this end, we compare contextualized models with their static counterparts. We make this comparison dependent on the application of a number of similarity measures and classifiers. Our results are threefold: Firstly, we show that the models combined with the different similarity measures differ greatly in terms of the amount of knowledge they allow for extracting. Secondly, our results suggest that similarity measures…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Natural Language Processing Techniques · Semantic Web and Ontologies
