[With apologies to Putnam, Pope, and all of you]
Two children are reading a text written by an AI:
The boys splashed water in each other’s faces until they were both sopping wet
One child says to the other “Wow! After reading some text, the AI understands what water is!”
The second child says “It doesn’t really understand.”
The first child says “Sure it does! It understands that water is the sort of substance that splashes. It understands that people who are splashed with water get wet. What else is left to understand?”
The second child says “All it understands is relationships between words. None of the words connect to reality. It doesn’t have any internal concept of what water looks like or how it feels to be wet. Only that the letters W-A-T-E-R, when appearing near the letters S-P-L-A-S-H bear a certain statistical relationship to the letters W-E-T.”
The first child starts to cry.
Two chemists are watching the children argue with each other. The first chemist says “Wow! After seeing an AI, these kids can debate the nature of water!”
The second chemist says “Ironic, isn’t it? After all, the children themselves don’t understand what water is! Water is two hydrogen atoms plus one oxygen atom, and neither of them know!”
The first chemist answers “Come on. The child knows enough about water to say she understands it. She knows what it looks like. She knows what it tastes like. That’s pretty much the basics of water.”
The second chemist answers “Those are just relationships between pieces of sense-data. The child knows that (visual perception of clear shiny thing) = (tactile perception of cold wetness) = (gustatory perception of refreshingness). And she can predict statistical relationships – like, if she sees someone throw a bucket of (visual perception of clear shiny thing) at her, she will soon feel (tactile perception of cold miserable sopping wetness). She uses the word “water” as a concept-hook that links all of these relationships together and makes predicting the world much easier. But no matter how well she masters these facts, she can never connect them to H2O or any other real chemical facts about the world beyond mere sense-data.”
The first chemist says “Maybe she knows things like that water makes iron rust. That’s a chemical fact.”
The second chemist says “No, she knows that (clear shiny appearance + wetness + refreshment) makes (dull metallic appearance + hardness) get (patchy redness). She doesn’t know that H2O + Fe = iron oxides. She knows many statistical relationships between sense-data, but none of them ever connect to the deeper chemical reality.”
The first chemist says “Then on what level can we be said to understand water ourselves? After all, no doubt there are deeper things going on than chemical reactions – quantum fields, superstrings, levels even deeper than those. All we know are some statistical relationships that must hold true, despite whatever those things may be.”
Two angels are watching the chemists argue with each other. The first angel says “Wow! After seeing the relationship between the sensory and atomic-scale worlds, these chemists have realized that there are levels of understanding humans are incapable of accessing.”
The second angel says “They haven’t truly realized it. They’re just abstracting over levels of relationship between the physical world and their internal thought-forms in a mechanical way. They have no concept of 
or 
. You can’t even express it in their language!”
The first angel says “Yes, but when they use placeholder words like ‘levels even deeper than those’, those placeholders will have the same statistical relationship with the connection between models and reality as 
.”
“Yes, which is the difference between being able to respond to ‘Marco!’ by shouting ‘Polo!” vs. a deep historical understanding of Europe-Orient trade relations in the Middle Ages. If all you know is that some statistical models are isomorphic to other models and to Creation itself, you still won’t have the slightest idea what the 
s of any of them are.”
“I’m not claiming humans really know what anything means,” said the first angel. “Just that it’s impressive you can get that far by manipulating a purely symbolic mental language made of sense-data-derived thought-forms with no connection to real 
at all.”
“I guess that is kind of impressive,” said the second angel. “For humans.”
God sits in the highest heaven, alone.
“Wow!” He thinks to Himself, “that cellular automaton sure is producing some pretty patterns today. I wonder what it will do next!”