Dear mNLP students,
Tomorrow at 15.00 in the Seminarraum 2 (ground floor) of the CAIDAS building Marco Bombieri (postdoc at Uni Verona and a visiting researcher at Uni Mannheim) will give a talk "Do LLMs Dream of Ontologies?". The talk is open for everyone and may be particularly interesting to those with deeper interest in NLP.
You can find the short bio of the speaker and abstract of the talk below.
Best,
Goran
--
Speaker: Marco Bombieri received his Ph.D. in Computer Science from the University of Verona, Italy, in 2023, as part of the E.R.C. Project "ARS" (Autonomous Robotic Surgery) at the Altair Robotics Lab. Since 2024, he has been a Postdoctoral Researcher at the University of Verona, focusing on large language models and knowledge representation. He is currently a visiting researcher at the University of Mannheim, Germany.
Title: Do LLMs Dream of Ontologies?
Abstract: Large Language Models (LLMs) have demonstrated remarkable performance across diverse natural language processing tasks, yet their ability to memorize structured knowledge remains underexplored. We investigate the extent to which general-purpose pre-trained LLMs retain and correctly reproduce concept identifier (ID)–label associations from publicly available ontologies. We conduct a systematic evaluation across multiple ontological resources, including the Gene Ontology, Uberon, Wikidata, and ICD-10, using LLMs such as PYTHIA-12B, GEMINI-1.5-FLASH, GPT-3.5, and GPT-4. Our findings reveal that only a small fraction of ontological concepts is accurately memorized, with GPT-4 demonstrating the highest performance. To understand why certain concepts are memorized more effectively than others, we analyze the relationship between memorization accuracy and concept popularity on the Web. Our results indicate a strong correlation between the frequency of a concept’s occurrence online and the likelihood of accurately retrieving its ID from the label. This suggests that LLMs primarily acquire such knowledge through indirect textual exposure rather than directly from structured ontological resources. Furthermore, we introduce new metrics to quantify prediction invariance, demonstrating that the stability of model responses across variations in prompt language and temperature settings can serve as a proxy for estimating memorization robustness.