Settings

Theme

Gemini 3 Pro vs. 2.5 Pro in Pokemon Crystal

blog.jcz.dev

315 points by alphabetting · 96 comments

Reader
15 threads
orbital-decay

The baked-in assumptions observation is basically the opposite of the impression I get after watching Gemini 3's CoT. With the maximum reasoning effort it's able to break out of the wrong route by rethinking the strategy. For example I gave it an onion address without the .onion part, and told it to figure out what this string means. All reasoning models including Gemini 2.5 and 3 assume it's a puzzle or a cipher (because they're trained on those) and start endlessly applying different algorithms to no avail. Gemini 3 Pro is the only model that can break the initial assumption after running out of ideas ("Wait, the user said it's just a string, what if it's NOT obfuscated"), and correctly identify the string as an onion address. My guess is they trained it on simulations to enforce the anti-jailbreaking commands injected by the Model Armor, as its CoT is incredibly paranoid at times. I could be wrong, of course.

  • jug

    I've had some weird "thinking outside the box" behavior like this. I once asked 3 Pro what Ozzy Osbourne is up to. The CoT was a journey, I can tell you! It's not in its training data that he actually passed away. It did know he was planning a tour though. It had a real struggle trying to consolidate "suspicious search results" and even questioned whether it was fake news, or running against a simulation!, determining it wasn't going to fall for my "test".

    It did ultimately decide Ozzy was alive. I pushed back on that, and it instantly corrected itself and partially blamed my query "what is he up to" for being formulated as if he was alive.

bbondo

1.88 billion tokens * $12 / 1M tokens (output) suggests a total cost of $22,560 to solve the game with Gemini 3 Pro?

  • mkoubaa

    I can't believe how massively underpaid I was when I was 11

    • re-thc

      Do you hallucinate as a kid?

      • foundddit

        At that age, it's called "imagination"

      • mkoubaa

        All kids confidently state incorrect things it's part of growing up

      • nomel

        Kids definitely do this. They fill in blanks/context with assumptions, resulting in all sorts of silly responses, for topics of sparse knowledge/certainty. They're not lying, because they think it's true. Sometimes the gap filling is wrong, but usually downright brilliant, within the context of their knowledge.

        • johnebgd

          Are you sure there is an age limit for that kind of behavior in humans?

          • nomel

            I replied within the context provided rather than all possible contexts. Would you also like to bring up the interpolation and extrapolation seen during problem solving in the cuddle fish, since it's related?

          • Aeglaecia

            calcification is as inevitable as entropy

      • anal_reactor

        My friend's son says he sometimes closes his eyes, imagines cartoons, and watches them.

  • elephanlemon

    “Gemini 3 Pro was often overloaded, which produced long spans of downtime that 2.5 Pro experienced much less often”

    I was unclear if this meant that the API was overloaded or if he was on a subscription plan and had hit his limit for the moment. Although I think that the Gemini plans just use weekly limits, so I guess it must be API.

    • bjackman

      Geminii CLI has a specific "model is overloaded" error message which is distinct from "you're out of quota" so I suspect whatever tools they're using for this probably have something similar, and they're referring to that.

  • brianwawok

    True though I bet the $200 a month plan could do it, maybe a few extra days of downtime when quota was maxed

    • AstroBen

      For how long would it stay $200 of you can rack up 5 figures if usage..

      • manmal

        That is the reason they severely limited Claude Max subscriptions. Some users racked up 1k+ in API equivalent cost per day.

    • jchw

      This is exactly why I upgrade to the Pixel 10 Pro. On Black Friday, you could get a Pixel 10 Pro for about $450 on the U.S. Google Fi store (which sells unlocked phones)... which is also about how much a Pixel 9 Pro goes for on eBay; minus eBay fees and accounting for shipping, that's an upgrade for < $100. But, it's even better, the Pixel 10 Pro comes with a year of their "AI Pro" plan (which I believe costs around $240/year.) There is really, really no point in upgrading to a Pixel 10 Pro from a Pixel 9 Pro, and environmentally it pains me to be the person upgrading my phone on an annual basis (this is the fastest I've ever upgraded a phone, ever) but it's hard to turn down when Google is selling $800~ish for $400~ish.

      And yeah, it's not the insanely priced AI Ultra plan, but if there are any hard limits on Gemini Pro usage I haven't found them. I have played a lot with really long Antigravity sessions to try to figure out what this thing is good for, and it seems like it will pretty much sit there and run all day. (And I can't really blame anyone for still remaining mad about AI to be completely honest, but the technology is too neat by this point to just completely ignore it.)

      Seeing as Google is still giving away a bunch of free access, I'm guessing they're still in the ultra-cash-burning phase of things. My hope (hopium, realistically) is that by the time all of the cash burning is over, there will be open-weight local models that are striking near where Gemini 3 Pro strikes today. It doesn't have to be as good, getting nearby on hardware consumers can afford would be awesome.

      But I'm not holding my breath, so let's hope the cash burning continues for a few years.

      (There is, of course, the other way to look at it, which is that looking at the pricing per token may not tell the whole story. Given that Google is running their own data centers, it's possible the economic proposition isn't as bad as it looks. OTOH, it's also possible it is worse than it looks, if they happen to be selling tokens at a loss... but I quite doubt it, given they are currently SOTA and can charge a premium.)

  • addaon

    To beat it, not to solve it. Solving means something very specific in the context of games — deriving and proving a GTO strategy.

  • ogogmad

    :/ Damn. That needs to cost 1000x less before people can try it on their own games.

    • someperson

      That's an extrapolation to finish the entire game.

      If limit your token count to a fraction of 2 billion tokens, you can try it on your own game, and of course have it complete a shorter fraction of the game.

  • echelon

    Who is paying for this?

    Did the streamer get subsidized by Google?

    (The stream isn't run by Google themselves, is it?)

    • emp17344

      If you go to the X page linked on the blog, the page owner mentions a “collaboration” with Google Deepmind on this project. It wouldn’t shock me if this just an elaborate advertisement for Gemini.

oceansky

"Crucially, it tells the agent not to rely on its internal training data (which might be hallucinated or refer to a different version of the game) but to ground its knowledge in what it observes. "

Does this even have any effect?

  • ragibson

    Yes, at least to some extent. The author mentions that the base model knows the answer to the switch puzzle but does not execute it properly here.

    "It is worth noting that the instruction to "ignore internal knowledge" played a role here. In cases like the shutters puzzle, the model did seem to suppress its training data. I verified this by chatting with the model separately on AI Studio; when asked directly multiple times, it gave the correct solution significantly more often than not. This suggests that the system prompt can indeed mask pre-trained knowledge to facilitate genuine discovery."

    • hypron

      My issue with this is that the LLM could just be roleplaying that it doesn't know.

      • jdiff

        Of course it is. It's not capable of actually forgetting or suppressing its training data. It's just double checking rather than assuming because of the prompt. Roleplaying is exactly what it's doing. At any point, it may stop doing that and spit out an answer solely based on training data.

        It's a big part of why search overview summaries are so awful. Many times the answers are not grounded in the material.

        • wavemode

          It may actually have the opposite effect - the instruction to not use prior knowledge may have been what caused Gemini 3 to assume incorrect details about how certain puzzles worked and get itself stuck for hours. It knew the right answer (from some game walkthrough in its training data), but intentionally went in a different direction in order to pretend that it didn't know. So, paradoxically, the results of the test end up worse than if the model truly didn't know.

      • stavros
        cg5280

        I like the inclusion of the graph at the end to compare progress. It would be cool to compare this directly to competing models (Claude, GPT, etc).

        • kqr

          It would unfortunately also need several runs of each to be reliable. There's nothing in TFA to indicate the results shown aren't to a large degree affected by random chance!

          (I do think from personal benchmarks that Gemini 3 is better for the reasons stated by the author, but a single run from each is not strong evidence.)

          • casey2

            TFA says multiple times that the results are affect by random chance

            • kqr

              Yes, but recognising that is only the first step. Quantifying the variance is the next step which I miss in the article.

        krige

        As a fun comparison, Gemini 3 Pro took 17 days to beat the game. Twitch Plays Pokemon, which was frequently random, chaotic, even malicious, took 13 days to clear Crystal.

        sussmannbaka

        So after years of being gleefully told that AI will replace all jobs an omniscient state of the art model, with heavy assistance, takes more than two weeks and thousands of dollars in tokens to do what child me did in a few days? Huh.

        • rybosome

          “And, because AI never got any better or any cheaper after that point, sussmanbaka’s wry observation remained true in perpetuity, forever.”

          - History, most likely

          • ehnto

            We will only be able to see where the economic chips lie once all the current money games shake out. It's all a bit obfuscated at the moment.

          • mchusma

            Cost per intelligence is shrinking by something like 100x per year. Even the Gemini flash release would potentially do as well for 1/5th already.

        • dwaltrip

          Children are incredibly smart. All of this was fantasy 15 years ago. Comments like yours are amazing to me…

        • murukesh_s

          I used to think the same until latest agents started adding perfectly fine features to a large existing react app with just basic input (in English) . Most of the jobs require levels of intelligence below that. It's just a matter of time before agents get to that.

          • blauditore

            It's about the complexity of the task. Front end apps tend do be much less complex and boilerplate-y than backends, hence AI tends to work better.

            • murukesh_s

              I disagree - having worked on backends most of the time, I find modern frontend much more complex (and difficult to test) than pure backend. When I say modern frontend - its mostly React, state management like Redux, Zustand, Router framework like React Router, a CSS framework like Tailwind and component framework like Shadcn. Not to mention different versions of React, different ways of managing state, animation/transitions etc. And on top of that the ever increasing complex quirks in the codebase still needed to be compatible with all the modern browsers and device sizes/orientation out there.

              • Jensson

                > When I say modern frontend - its mostly React, state management like Redux, Zustand, Router framework like React Router, a CSS framework like Tailwind and component framework like Shadcn

                AI is the best at adding standard things into standard boilerplate situations, all those frameworks just makes it easier for AI. They also make it easier for humans once you know them and have seen examples, that is why they exist, once you know those frontend is not hard.

              • blauditore

                Actual complexity depends is not imposed by the framework, but by the application itself, and the amount of front-end logic tends to be lower. Yes, there is more complexity in the setup of front end code (now there are dependencies and build pipelines), but ultimately they should simplofy things especially for slighly more complex websites.

                Testing is one of the things that's generally tedious in front end applications, but not inherently complex. There may be lots of config needed (e.g. for setting up and controlling a headless browser), and long turnarounds because tests are slow and shaky. But they are also boilerplatey.

              • rafaelmn

                That's just a farmiliarity thing. I've worked on project doing full web FE, mobile and BE.

                It's hard to generalize but modern frontend is very good at isolating you from dealing with complex state machine states and you're dealing with single user/limited concurrency. It's usually easy to find all references/usecases for something.

                Most modern backend is building consistent distributed state machines, you need to cover all the edge cases, deal with concurrency, different clients/contracts etc. I would say getting BE right (beyond simple CRUD) is going to be hard for LLM simply because the context is usually wider and hard to compress/isolate.

                • murukesh_s

                  >Most modern backend is building consistent distributed state machines, you need to cover all the edge cases, deal with concurrency, different clients/contracts etc. I would say getting BE right (beyond simple CRUD) is going to be hard for LLM simply because the context is usually wider and hard to compress/isolate.

                  Seeing the kind of complexity that agents (not standalone llm) are able to navigate - I can only start to believe - just a matter of time it can do all kinds of programming, including state of the art backend programming - even writing a database on its own - good thing with backend is its easily testable and if there is documentation that a developer can read and comprehend - an llm/agent would be able to do that - not very far from today.