In 1950 Turing was asked whether machines could think. He declined to answer. He proposed the Imitation Game instead, on the grounds that we possess no private test for thought even in one another. The substitution has been resisted ever since, and large language models are making the resistance harder to sustain. We extend to one another the courtesy of assumed inwardness and withhold it from the machine, though we have no better evidence in our own case.
The standard objection, sharpened recently by Chomsky, is that such systems produce the right behaviour for the wrong reasons: fluency without competence, surface without the scaffolding that lets a child generalise from a sparse sample. Whatever they are doing, it is not what we are doing. The resemblance is a coincidence.
This is a serious objection, but it cannot be cashed out. To say a system fails to think because it lacks the right internal computation is to appeal to evidence no one can obtain. Every attribution of thought we make about another mind proceeds through behaviour, because behaviour is the only access we have.
Consider the institution that has tried hardest to reach inside other minds. Criminal law has spent its history assessing intent through inference (conduct and circumstance). Mens rea is not a window into the defendant’s mind. It is a disciplined procedure for reading behaviour and giving the reading an interior name. The inner test turns out, on inspection, to be a behavioural test in different dress. If this is how we handle the question when liberty is on the line, the demand for something stricter from a language model is doing work the evidence cannot support.
“Is it really thinking?” may turn out to be a question the concept can’t answer. In the nineteenth century people argued about whether a virus was really alive. The argument wasn’t silly, but biology didn’t settle it. Biology moved on, and at some point, the question stopped being the interesting one. Something like that seems likely here.
In the meantime, the systems work, and the question of what they are trails behind that fact. I did not design my brain, do not understand it, and rely on it constantly. I can’t inspect what it’s doing. Turing understood this in 1950. The machine is obliging the rest of us to catch up!