A practical guide to AI image generation, from diffusion to agents
danielball.comNice article. There's another option for pixel art that's recently been coming into vogue as tool usage improves in LLMs which is procedural graphics.
I actually experimented with creating a set of skills around rendering early AGI style EGA graphical scenes, and it worked surprisingly well.
So rather than storing full bitmaps, you instead store sequences of simple drawing instructions designed to recreate some specific scene - like you would an SVG.
As a result you end up sidestepping a lot of the problems with larger generative models, which can have issues with grid alignment, palette choices, and high-frequency noise.
Here AGI means Adventure Game Interpreter.
Exactly, turn something nondeterministic into something more deterministic and AI does a great job at that.
This is very cool! I will try your method for background and tile creation in the asset-generation tool I've built [1]. Would your procedural method work for animations as well, like sprites?
Looks great. This could be a form of image compression.
Several Amiga/DOS era games used a similar system for image compression.
I suspect your picscript language is an entirely unnecessary abstraction and the prompt (ie the model) is doing all the work.
How so? Judging from the examples the model does seem to take advantage of the language (creates and uses reusable elements) and AFAICT the language itself compiles down to "real" Sierra PIC commands so in theory it can be used with a Sierra PIC viewer/engine.
Yeah compiling to Sierra PIC is a unique advantage over asking the model for an image. But the claim im attacking is different. The claim in attacking is that you cant just tell it to produce images like this directly.
>As a result you end up sidestepping a lot of the problems with larger generative models, which can have issues with grid alignment, palette choices, and high-frequency noise.
All of these can be avoided with simple prompts and don’t require all this ceremony.
Convo then the artifact from the convo (sorry, apparently no way to share the pics but they are just captured from the artifact).
Convo: https://claude.ai/share/78cd63c8-ffdd-4569-8de2-ac8ab5c5cfe0
the style is completly differrent…
How so?
The difference i see is the shading. Could easily tell it to do flat fills instead.
Regardless it has none of the supposed problems of AI generated pixel art.
Neat! I've followed you on Github. I like the idea of going to the past to see what works and adapting it for the LLM age. Nice work!
Thanks! Yeah, I started building it because I read an interview with the original creators of King’s Quest, Ken and Roberta Williams where they mentioned in passing that it was too expensive (in the 80s) from a storage perspective to store entire bitmaps for each scene, so instead they'd just store a series of very simple drawing instructions and render on demand.
I’m super excited to see more LLM-driven procedural graphics stuff in the future!
I think this is missing a nice hack.
Learn to draw.
Then ask the model to restyle the image.
You can get wonderful photographs with the exact composition and scene you want for all sorts of use cases.
Second hack.
Model the scene you want from low resolution geometry (boxes, spheres etc) for the composition. Then use that to ground the generated image (ie give it as reference) and ask the model to re-imagine it in the style you want.
I don't think this told me anything, and it reads like basically something I could have asked claude to whip up for me.
Which is half of these new websites these days.
These models always create pixel art that looks like garbage because the people using it don't know enough about the medium to properly judge the output. This article is no exception. If you want pixel art for your project then learn or hire someone with that expertise
Are you saying people unfamiliar with the 'pixel arts' can't enjoy it their own way?
I don't think the author is posing as a professional. These models are like having a 3D printer at home. Very handy in a pinch, but what they make doesn't replace the real thing.
> Are you saying people unfamiliar with the 'pixel arts' can't enjoy it their own way?
The problem isn't that they look shitty to pixel artists or experts, they look shitty for anyone with a bit of observation.
It's junk art the same way Mc Donald's is junk food.
It's calorie-dense and high in sodium?
I think we could do a bit better here with the explanation of what looks bad to you..
I am just a layperson when it comes to pixel art, but to me the example in the article is clearly garbage/low quality.
- the magenta background is still visible along the edges - the color palette is all over the place - random noise pixels of totally different colors are visible scattered around - the lighting and shading doesn't look that good
> It's calorie-dense and high in sodium?
There's plenty of real food that is as heavy and as salted, and as such no healthier, than junk food. But it actually tastes good. (I can't exactly put the finger on what makes an Italian pizza better than a US 7/11 one, but it's not about the former being healthier).
I'd say it's the fermented, charred dough in combination with fresh, varied toppings.
With the exception that it's harder to make the argument that "it's bad for the consumer."
McDonalds', over time, will make you less healthy.
AI slop over time will perhaps change aesthetics, but hard to chalk that up to anything other than personal preference.
But if the user thinks it looks fine, then won't most players think the same?
I just don't get the complaints about "use proper art" when it comes to games. Most of the time the game has some other, far more glaring, visual issues (anti-aliasing, clipping, icons-not-matching-items).
You don't want to anti-alias when having pixel art.
> But if the user thinks it looks fine, then won't most players think the same?
I have a suspicion that when you're looking at (and generating, because you then look at the results) AI generated images and text more you get desensitized and are more likely to say that something looks okay when to other people it won't.
Really? I'm noticing the opposite. First round of imagegen assets seem basically ok, but as time goes on you start to notice the similarities that are properties of the current crop of models and they start to annoy you.
People aren't using it wrong (although it's true most users have no taste), it's just the model devs don't care and generally have no idea about pixel art at all. The model just needs to be trained by someone knowledgeable. For example Retro Diffusion worked pretty well for its era - no mixels, proper palettes, etc.
As soon as artists stop blacklisting devs for using AI placeholders, we can get right on that.
Isn't that a good thing? Since these models generate garbage output of very low quality, they will never compete with the high quality generated only by human artists. Therefore this is a passing fad and human artists will always dominate.
The worst result would be if these models were any good vs being absolute slop. Then human artists would need to worry. But in fact there is no need to worry at all, as surely people (aside from a few people with bad taste) will clearly see the difference.
how will they tell that the person hired did a good job if they can't tell if the ai cocks up?
Seems like an interesting article but I can’t be bothered to wade through all the Claudish to get to it
> Use the same style suffix and seed for every asset in a category so the batch doesn’t drift.
This part's a little confusing to me. What does "suffix" mean in this context? (Is it referring to just like the style-instructions boilerplate text you might include with each image?)
And then also: does locking a set seed really have an effect one way or other for consistency? Wouldn't changing the prompt text even if only slightly have just as much a randomization effect as changing the seed, making fixing it or using a random seed both functionally the same? Or is that not how the latest models work?
All of that still reads as a non-professional messing about at ten different things.
If we want to use a pixel art generator commercially we need 'practical guide' to making a small change to one animation, two months after creating the original character.
We need images generated in psd layers, with actual text using a font.
Sticking to the brand palette religiously.
My guess would be that "professional" essentially translates to consistent, reliable and structured in the fashion a given industry expects. In this sense, everything AI/LLMs do is going to be "non-professional". Even when they solve frontier level math problems, the mathematicians agree "this is a terrible proof" (this goes against how the field does things). These things is always inconsistent/unreliable/unstructured but sometimes not too inconsistent etc to be useful. "How do use" tip sheets are also going to a list of ad-hoc observations, many of which its easy to get after X many tries with the generator - but that OK, essentially what you want is ways to save yourself time screwing around.
These thing may reach your standard consistent at some point, maybe. It's not important since before and after this point, they will require "messing around" to produce something useful but they can produce something if you need "good enough" but if you demand actually good, they won't arrive ever.
Great article -- and maybe this is just me -- but I do find the primacy of the online services interesting. Me and my 16gb vram card can get most of this done at home for "free."
When your work depends on people using online LLM services, you probably don't want to encourage people to use their own hardware.
Are control nets not being used much anymore? I haven't seen anything that provides a similar level of composition control, and yet I do not see them being referenced much at all.
They are used a lot, and modern edit models kind of support base cases - depth, canny, openpose - as a reference image. I mean they understand "use image 2 as depth map", result indeed have same depth, etc. But interpretation sometimes arbitrary to model.
For the tasks where several control nets should be adhered (depth and canny and segmentation) dreaded SDXL is still unmatched. SDXL also the only one handling influence weight for each control net normally, this ability completely vanished on modern models
Control nets in early post-SDXL models like ZIT or Flux kind of work too, but still much worse in quality when combined. I suspect pure diffusion architecture of SDXL is really favorable to mixing control signals with precision, DITs probably can be trained to similar level for mixing signals - but it is much harder and never seen in practice. At least every post-SDXL model I tried is trained with single one in mind, judging from my attempts. This rant is from personal expirience.
Related question: Whenever I try to run Image generation locally, be it with Flux or Qwen, I seem to be getting results that are about 1-2 years behind what ChatGPT can produce today - and take up to 5 minutes for a single image. Is this a hardware problem (running it on a decent windows gaming PC), a configuration problem or just the reality of what local models can deliver at the moment on consumer hardware?
If you're on a GPU and take five minutes for an image, something is wrong: either you're not actually running on GPU, or you're generating at way too high resolution. Open models work best at not far above 1024x1024.
> Whenever I try to run Image generation locally, be it with Flux or Qwen, I seem to be getting results that are about 1-2 years behind what ChatGPT can produce today.
At the risk of stating the obvious, you can't really compare local models to proprietary frontier models. For all we know, those models are heavily subsidized and have trillions more parameters than the smaller, modest ones you can run on a single GPU.
I've been running a prompt-adherence benchmark site [1] all the way back since the days of SDXL. If you look at the progress dropdown, I do think the gap is closing.
June of this year saw the release of some of the biggest open-weight bangers (Krea 2, Ideogram 4, and the kind of slept-on Boogu-Image 0.1), though the license terms can be kind of meh.
The best locally hosted model scores 8 out of 15, while the proprietary ones score 12 out of 15 on the text-to-image metric. If you want something that can reasonably decent for local gen while still being relatively small for modest GPUs, take a look at the recently released Moss-Image model [2]. But if you're expecting results at a level of something like GPT-Image-2.5 Sunburst, you're going to be disappointed.
Thank you for this, I'm a bit behind the times now on local generations and tend to use nano banana pro now when i generate although i used to use stable diffusion and automatic1111 way back, dabbled in everything around and Midjourney 3 onwards. The website shows me clearly where things are at. I expect I'd still like playing with Midjourney for more artistic styled work but it was the cost that drove me away.
I found that for Qwen Image 2.1 the prompt enhancer makes a big difference.
It won't bridge the gap to gpt-image-2.5 though, and it's going to slow down the process further.
With a 4090, Qwen Image 2.1 Turbo takes around 15s per image, depending on the resolution. That's without the prompt enhancer.
With my gamer 4060 ti 16gb it takes around 20 or 30 seconds for flux 1024x1024 image. At least that's how I remember it.
Configuration and maybe prompting problem in my experience. Are you doing the prompting or are you letting claude or something do it?
Look into r/StableDiffusion and follow their prompts and download their ComfyUI workflows.
Thanks, I've done that, but with mixed results. I've also tried comfyui wrappers and alternatives that try to hide the complexity and just set sensible defaults, but I'm beginning to suspect its ultimately a consumer hardware vs million dollar NVidia server question.
What are you trying to generate? I've also noticed that local models aren't as good as cloud models due to hardware but there are still ways of optimizing the process.
You can get results a billion times better than the horrible, disgusting look of ChatGPT in less than a couple of seconds with a model like Krea or MOSS, and ComfyUI. With just a 8GB VRAM GPU or so...
Just visit Civitai, look at the pages for the base models, filter for images with the ComfyUI workflow in them, then you can just download and drag and drop them onto your local ComfyUI app.
SwarmUI makes things a lot easier. Kroma generates an image in about 90s with a 4070.
I have been asking opus 5.5 to write python to create logos. It is more consistent and easier to control than gen AI for this purpose
I've been using it to generate animated svgs. I made a breakthrough in asking it to create bones/rigging and movement with a python library and apply the result to the svg.
How much aesthetic freedom are you giving to opus?
The technical ability of anthropic models continue to steadily improve, but I still find the creative and compositional choices to fall far short of anything I would ever use.
Good content. Found it helpful
You mentioned in your README that it was tested with GLM-5.3 but this is not a multi-modal model. Did you mean GLM-5.3-Flash?
Claude, turn this into a skill.
One of the better TLDR's I've seen recently.