Ethan Mollick (@emollick) on X

Post

Post

• 

  So the suspicions about the dumbing-down of GPT-4 may actually be right! Here is some initial hard evidence that GPT-4 is actually getting less capable (and GPT-3.5 is getting more so), since launch. Also, why it is hard to build on AI, when model abilities are quietly changed.

  

  

  Lots of people are wondering whether #GPT4 and #ChatGPT's performance has been changing over time, so Lingjiao Chen,

  @james_y_zou

  and I measured it. We found big changes including some large decreases in some problem-solving tasks: arxiv.org/pdf/2307.09009…

  

• 

  The paper doesn’t get at why the degradation in abilities is happening. We don’t even know if OpenAI knows this is occuring.

  

  I also think it is possible this is unintentional or a consequence of further alignment. And there maybe tasks where ut has improved. We don’t know.

  

  Replying to @random_walker

  This from a VP at OpenAI is from a few days ago. I wonder if degradation on some tasks can happen simply as an unintended consequence of fine tuning (as opposed to messing with the mixture-of-experts setup in order to save costs, as has been speculated). x.com/npew/status/16…

  

  

  Replying to @random_walker

  OK, I re-read the paper. I'm convinced that the degradations reported are somewhat peculiar to the authors' task selection and evaluation method and can easily result from fine tuning rather than intentional cost saving. I suspect this paper will be widely misinterpreted.😬

• 

  This paper is really interesting. I actually went to the playground and tested the March version against the June one on similar tasks, and I was able to replicate all of their cases. I suppose I'll stick to version 0314 for coding until I gain more clarity on the matter.