Sam: I call it the Addomatic! You speak two numbers into this microphone, and in less than half a jiffy it prints out their sum on this tape!
Gary: I’ll have to see it in action before I believe you, of course.
Sam: Certainly!
(Sam leans into the microphone and articulates carefully.)
THIRTY-EIGHT. FIFTY-FOUR.
(The machine whirrs and beeps before ejecting a thin strip of paper, which Sam tears off.)
Aha, 92! Neat, huh?
Gary: May I try?
Sam: Certainly, please do!
(Sam turns the machine’s microphone towards Gary.)
Gary: SIX HUNDRED AND THIRTY-THREE. SEVEN HUNDRED AND FORTY-NINE.
(The machine whirrs and beeps. Gary tears off the paper slip.)
Oh, 1472. That's not right, the answer's 1382!
Sam: Well it does go a bit wobbly above two digits, but it got the first and last digits right!
Gary: A bit wobbly? Sam, you can't be serious! You can't go around calling this the “Addomatic” — it can't add! It can't add three digit numbers! Whatever mechanisms you’ve employed in this machine of yours are clearly unsuitable. You need to start over. You should probably read my 2001 book ...
(One month later…)
Sam: Behold, the Addomatic Two Thousand! Shift those empty mugs, Gary, I need to set this thing down. It’s quite a bit heavier than the previous model.
Gary: (sarcastically) Certainly, Sam, certainly.
Sam: Now watch this. SIX HUNDRED AND THIRTY-THREE. SEVEN HUNDRED AND FORTY-NINE.
(The machine whirrs, beeps and clunks, for longer now, before extruding the familiar paper slip.)
Gary: From the look on your face, I’m guessing the printout says 1382?
Sam: (satisfied) It does indeed.
Gary: Very nice! I take it my 2001 book got you on the right track? Why, you practically named the new model …
Sam: No, actually. You see the mechanisms you so derided last month were in fact perfectly adequate. They simply needed to be scaled up.
Gary: Quite substantially, I see. But Sam, I note we should really try it on something new. Any Tom, Dick or Harry could construct a machine that always spits out "1382" whatever goes in. May I?
Sam: Be my guest! It’s been through substantial evaluations and hasn’t had problems with …
Gary: EIGHTY-TWO QUINTILLION, FIVE HUNDRED AND NINETY-ONE QUADRILLION …
Sam: Well hold on a hot minute.
Gary: What’s the matter?
Sam: Well, well that’s a big number there, Gary. Quite a big number.
Gary: It’s called the Addomatic Two Thousand. It can add, can it not?
Sam: Gary, Gary. You've moved your goal posts. Your complaint last month was about three digit numbers. It can do those now! And the Addomatic Three Thousand, which is only halfway built mind you, is already up to ten digits!
Gary: No, you misunderstood me. To be able to add means being able to add arbitrarily large numbers. The only reason I focussed on three digit numbers was to illustrate how hopelessly far from truly being able to add that machine was. Your hesitance to let me give the new model a proper run leads me to believe it may have the same fundamental limitation.
Sam: Last month you literally said "It can't do three digit numbers so the whole method is doomed", and now it can do three digit numbers! Make it make sense, Gary!
Gary: Not being able to add three digit numbers shows you can't add in general, and throws into doubt your whole method. But the reverse does not hold! Being able to add three digit numbers does not mean you've learned some general adding method that works for arbitrarily large numbers.
Sam: By those standards I could never prove to you that the machine can "truly" add, because there could always be a pair of numbers I haven't tried yet where it fails. So isn't your claim that the machine can't add unfalsifiable?
Gary: No. If you explain how the machine works, you might be able to prove to me that it follows a fool-proof algorithm that gets the correct answer no matter how large the numbers get. That would falsify my claim.
Sam: But I don't know how it works! The machine is just a massive network of wires I connected up literally at random into a huge electronic circuit! Each wire has a little dial controlling its resistance, and I just tuned each one in turn until the machine started printing out the correct answers.
(At this point, Gary’s student, Luna, who has been listening in, interjects.)
Luna: But surely you know how the final circuit works, otherwise how did you know how to set the dials?
Sam: Trial and error!
Luna: But that would take literally forever! You can't just try random settings until it works. The chance of a random circuit correctly converting sound recordings of numbers to dots on a slip of paper denoting their sum is one in a gazillion bajillion. You'd be twiddling dials until the heat death of the universe!
Sam: (cunningly) The trick, Luna, is not to twiddle the dials at random, but to tune them very slowly, bit by bit. You play recordings into the machine one by one, tweaking the dials after each round until the machine improves.
Luna: But how do you know which way to tweak the dial for each wire? Suppose you play into it the recording "Five. Two." and it prints out four. If you don't know how the circuit works, how can you possibly tweak it to make it print out seven instead?"
Sam: Oh, I forgot! The print out thingy is just an add-on.
(Sam unclips a hatch on the machine’s back to reveal a large gauge, going from one to a million.)
The circuit actually controls this gauge here. Then this widget does the final step of reading the gauge and printing out the number. The widget is just simple software of course, not intelligent in the slightest.
Gary: We agree on that, at least.
Sam: Anyway, the widget is added afterwards — it's this gauge that I watch when I tweak the dials.
Luna: Oh I see! So even though you can't tell what each particular wire does, you can see whether tweaking its resistance up or down moves the gauge towards or away from the correct answer?
Sam: Precisely! And once you've done this for all the wires, you play another recording and repeat the whole process. After around, I don’t know, 100,000,000 or so recordings, the gauge more or less always points to the correct answer. Then you attach the printing widget and hey presto, the Addomatic!
Gary: 100,000,000 recordings and it still couldn't do three digit addition! Ooooh, AGI's coming guys, watch out guys I'm real serious! Help! Senator Blumenthal, heeeeeelp!!
Sam: Gary, it’s not as though you disagreed.
Gary: Heeelp, Blumenthaaaaal! Significant harm! Quick! Regulate me!! Hahaha, good times, good times!
Luna: You were saying something about falsifiability before I interrupted?
Sam: Yes, let's get back to the point, Gary. How could I ever falsify your claim that my machine can't add?
Gary: We can worry about that once I can't find examples where it gets it wrong.
(Gary leans into the microphone)
EIGHTY-TWO QUINTILLION, FIVE HUNDRED AND … TWO HUNDRED AND ELEVEN. SEVEN TRILLION, FOUR HUNDRED AND … AND FORTY-SIX.
(The machine whirrs, beeps and clunks. Gary takes out his pocket calculator.)
Sam: What do you need that for, Gary? Can't you add?
Gary: I don't have all day, Sam.
(Gary starts punching numbers into the calculator before realising it only goes up to nine digits. He reaches for his laptop and opens up a Python terminal.)
Sam: Sure looks like your calculator can't add either, Gary. Why, it trips up once the numbers get too large.
Gary: Don't kid yourself, Sam. Unlike your so-called Addometer, we know the calculator can add because we know how it works! We designed it to manipulate symbols according to a fool-proof algorithm.
Sam: You literally said earlier that being able to add means you can add arbitrarily large numbers, and it can't! By your lights, the calculator can't add! That's a reductio ad absurdum if I ever saw one.
Gary: Suppose I create a giant table of all possible pairs of numbers up to nine digits and their sums. Then I create a "calculator" where you punch in the numbers, it looks up the answer in the table and displays the answer on a screen. Are you telling me you don't see a difference between such a machine and my pocket calculator?
Sam: Functionally, no! If it's always correct either way, what do I care how it does it?
Gary: You should care because which one it is tells you whether you're onto something or being led down the garden path. If Charles Babbage's difference engine had turned out to be a giant lookup table, people would have been rightly unimpressed. The fact that he did it without a lookup table demonstrated that a new kind of machine had been built, one that could manipulate symbols according to algorithms. Algorithms you can prove give you the right answer every time.
Sam: Fine, but how do you know my machine isn't implementing a fool-proof algorithm?
Gary: Because it makes mistakes, Sam, we've been through this!
Sam: But so do humans! Your bar is too high.
Gary: There's a difference between making the odd mistake when following a fool-proof algorithm, as humans do, and relying on surface-level pattern matching without any true understanding, as your machine does.
Sam: Well then I simply ask again, how do you know how my machine works? How do you know it doesn't follow a fool-proof algorithm with the odd mistake? Good grief, I built it and I still don't understand how it works. How can you? Why is the odd mistake excusable for a human, but apparently a refutation of my machine?
Gary: Because we can introspect and watch ourselves following algorithms by manipulating symbols. For example, when you add 17 and 26, you can notice yourself carrying the 1. I’ve no reason to believe your machine is doing this rather than pattern-matching from the hundreds of millions of recordings it has been trained on. Following algorithms by manipulating symbols is a very sophisticated capacity! If someone comes to you with a machine they themselves don't understand, you are quite justified in being highly skeptical that it has such a sophisticated capacity.
Sam: Sure, I get that you should start sceptical, but you also have significant evidence in favour, no? The Addometer Three Thousand reliably adds up to ten digit numbers! Are you unmoved by that?
Gary: I am not unmoved! I just think a much simpler and therefore more likely explanation is that it's doing surface-level pattern matching. The idea that it magically learned how to manipulate symbols according to algorithms, having never been taught? It’s rather fanciful.
Sam: It wouldn't be magic if it learned symbol manipulation. Both surface-level pattern-matching and true algorithm following are reinforced when tweaking the dials. Remember, I tweak the dials without knowing what each wire or sub-circuit does. If one sub-circuit just so happens to do some relevant symbol manipulation, albeit badly, it will be reinforced. You don't need to be explicitly taught how to manipulate symbols in order to learn how to manipulate symbols.
Gary: You really ought to read my latest blog post, where I explain how... Wait, say that again?
Sam: What I'm saying is that the number of wires I put inside the machine is insane, like hundreds of billions, so it's not unreasonable to think that even before the dials have been tweaked — when they're still all set up randomly — that some parts of the enormous network will by chance be doing a kind of symbol manipulation.
Gary: Well those sub-circuits aren't going to be much use if they're not hooked up in the right way. They need the right connection to the input, each other, and the gauge that's meant to show the correct answer.
Sam: True! But suppose some of the wires are able to connect these sub-circuits up in the right way, if only their dials are set correctly. Then when I'm tweaking these wires, they will have a big effect on bringing the gauge to the correct value. Maybe, when I’m tweaking the wires, I'm mostly just connecting up sub-circuits in the right way to build up a big symbol manipulating circuit!
Gary: Sounds a bit far-fetched. Aren't you also building a Speak-o-matic that answers questions? What’s it called, Chappatiti?
Sam: Ah, the new machine. It will change the world forever. I'm honoured to be bringing this new type of computer to humanity, which will improve countless lives ...
Gary: Chappatuto?
Sam: ... and drive us gently towards the singularity. Of course, it is not without risks, but we are working responsibly ...
Gary: I'll stick with Speak-o-matic. I can just about see your hypothesis that by tweaking the dials you're mostly connecting up useful sub-circuits that happened to be there at the beginning. Fine. But for language? I suppose you think that even before the dial-tweaking there was, by chance, a sub-circuit for telling you capitals, a sub-circuit for recipes, a sub-circuit for translating French verbs etc. The odds that for every pattern of speech there would — by chance — already be a sub-circuit ready to be connected up? Why Sam, it beggars belief!
Sam: No, that would indeed be ridiculous. Those kinds of sub-circuits are far too sophisticated to occur by chance. I'm thinking of much more basic components.
Gary: Such as?
Sam: Well, the Speak-o-matic works by always predicting the next word, right? Maybe right from the beginning there's a very simple sub-circuit which, when fed a recording starting "Mr Smith went for a walk with Miss Taylor. When he grew hungry, Mr..." the sub-circuit completes it with "Smith."
Gary: And then another sub-circuit for Mr Williams, another for Mr Brown, another for...
Sam: No, a single sub-circuit that, when fed recordings starting "Mr X blah blah blah. Mr ..." completes with "X", whether that's Smith, Williams or Brown etc.
Gary: Sounds quite unlikely that this would occur by chance!
Sam: That might be why I had to put hundreds of billions of wires in, so that at least some sub-circuits like this would be there from the start.
Gary: Fine. But language is so rich! You're going to need thousands, nay millions of these sub-circuits. And the chance of all of them being there at the start? You've got to be kidding? Besides, even if the sub-circuit you described worked for Mr Smith, Mr Williams, Mr Brown, Mr whoever, that's still a surface-level pattern in my book.
Sam: My guess is that these millions of necessary sub-circuits don't all need to be there from the start. As you tweak some circuits, that has downstream effects on how others behave. So as you're tweaking, parts of the network that were previously garbage (because they were receiving garbage) now happen to do the beginnings of something useful, which can then be reinforced and refined by further tweaking. Also, as you're tweaking, you sometimes tweak things in basically random ways because of noise in the recordings that have spurious effects. That’s another way in which further useful sub-circuits could appear while you're tweaking, and...
Gary: Sam, this is all very speculative. Let's bring things back to the Addomatic?
Sam: It's not all speculative!
Gary: Well, I at least see now that the Addomatic could in principle learn to implement symbol-manipulating algorithms without explicitly being taught, and that this doesn't require any kind of magic. It just seems very unlikely, that’s all. And I'm completely content dismissing it as a remote possibility, with the onus being on you to prove that's what's actually happening inside your machine.
Sam: That's fair enough, but you do see that's quite a way away from what you originally claimed?
Gary: PEOPLE ARE ALWAYS CLAIMING I SHIFT MY VIEW WITHOUT ACKNOWLEDGING IT BUT IT'S NOT TRUE THEY SHOULD READ MY 1976 BOOK I'VE BEEN THINKING ABOUT THIS SINCE BEFORE YOU WERE BORN I'VE ALWAYS BEEN VERY CLEAR I EXPECTED BETTER FROM YOU I ...
Sam: (arms swishing) The totality of your public statements is logically consistent!
Gary: (gasping): Hhhhhnnnnhh!! Uuuugghhh...
Sam: Crikey, are you okay?
Gary: (furious) Never do that again.
Sam: No kidding!
Gary: Last time somebody did that I wrote thirteen blog posts, two Guardian articles, and seven hundred and thirty-two tweets.
Addomatic 2000: WHIRR, BEEP, CLUNK, SKRRT, HISSSSSSS …
(Silence)
Addomatic 2000: BANG!
Sam: Shall we pick things up tomorrow? I know a paper you might find persuasive.