Another one of these posts that really don’t belong on the “AI fails” substack, but that’s the one I have. Maybe the name will just become ironic at some point…
I just tried giving ChatGPT 5.6 Sol a problem — really, a few variants of a mathematical problem — in my own research area that I was just wondering about (inspired by this little game), maybe not the world’s most exciting problem and related to other problems, but still, to my knowledge, a new problem that nobody ever worked on before and requires some real new work, and it is interesting to me at least. In a single shot, in 87m 23s, it produced a 24-page research paper from scratch that settles the main questions I asked about and then some, and finds some related research I didn’t know about.
What I’ve checked of it so far checks out. There are more things one could have done as it acknowledges, but if this had been an advanced undergraduate doing a semester-long independent study, I imagine it would have very easily been an A and I would have encouraged the student to work on it just a bit more to submit it for publication somewhere.
This was my 281-word prompt (most of which won’t make sense if you don’t know game theory). It hints at how I think it’ll probably turn out, but I’d do the same for the undergrad.
Consider the following problem. In Nash equilibrium there is the condition that every unplayed strategy is not a better response. What if we drop that condition and just say that only the played strategies must be best responses? This is equivalent to finding a subset of the strategies for each player such that the corresponding game (throwing away the other strategies) has a fully mixed Nash equilibrium. There would be a natural evolutionary game theory interpretation where the missing strategies just don't exist in the population and nothing can suddenly "mutate" into them. So let's call it a "no-mutation equilibrium" (NME). In this context, it's not even clear that certain questions in two-player zero-sum games are solvable in polynomial time. For any single strategy, that one together with any single strategy for the other player is an NME. But it's not obvious that problems such as "is the first row part of any NME that involves other rows" or "is there an NME that includes both of the first two rows" etc. can be solved in polynomial time. There are lots of potential variants -- symmetric games, extensive-form games, etc.; wanting the NME to be unique for the specific rows and columns chosen (should be true in generic games?); etc. Has this already been studied? If not, or if there are significant gaps, I want you to write an entire paper on this topic in latex, in response to this single prompt, in a similar style as other papers on similar computational problems. The paper should of course have references etc. Please work hard on this and make sure that any proofs you include are as simple and comprehensible as possible.