Words are lossy compressor, LMs speaks in meanings
ruikai.posthaven.comOnce the author started talking about LLMs having a "deeper understanding" I stopped reading.
This high dimensional vector space does allows them to theoretically embed a interpretable "deeper understanding", reference https://www.youtube.com/watch?v=wjZofJX0v4M&t=624s. It's the pre-train process that determines whether or not they demonstrate this "deeper understanding".
This "deeper understanding" refers to the understanding to the word, can be contextual, pragmatic, or semantic.