I experimented with LLM coding for a bit but quickly scaled back my usage to only niche use cases. I started by using it for data visualization and it one-shot converted my descriptions to the exact visuals in my mind. I then was dealing with a chunking edge case and it also solved it perfectly.
Only two days in and I found myself pulled towards a prompt every time I needed to think through a problem. It was addicting in a way that I’ve only ever experienced with gambling (with which I had a similarly brief foray). I soon found myself going back and forth, trying again and again to one-shot a more complex feature, going back and forth between the output and prompt specifications. I then closed the language model, spent 10 minutes of focused effort and built exactly what I needed.
Programming languages allow you to break down problems into multiple levels of hierarchy and abstraction. For me personally, even if I think through the approach, I only discover this structure after my first pass at a solution. I often tend to throw away my first solution and rewrite from scratch. Coding with LLMs for anything non-trivial hides this internal learning: I was instead operating at an input/output level.
If code is building blocks, LLMs can operate very well at the single block level. However, it’s not always obvious to me whether my current task is a single building block or a collection of them. Combined with the dopamine hit from successful one shot solutions and it was easy for me to turn the my slow and steady software engineering into a game of slots. Some huge wins but a net loss.
I’m sure a more discerning engineer could have a better intuition for the best time to hand off items to an LLM. But for me, I had to make a hard rule for myself that I won’t ever copy/paste LLM code to any of my work projects.