How to build a diffusion language model

I've been studying these a bunch for a project in university. Last week I went over the derivation of the ELBO for a couple hours and it was a very fun and elucidating exercise.

Once you give names to the larger mathematical structures and understand them a bit better it becomes quite simple. I wish some of the blogs/papers I'd read had named "Importance Sampling".

The probability notation can be pretty confusing too. Sometimes it's hard to understand the "types" of some variables. But I'm inexperienced.

ChatGPT was surprisingly helpful. If you put in the work to truly understand the where the gaps are in your mental model (which parts aren't completely intuitive), it can do an amazing job filling in the gaps.

A good video from welch labs on image generation with diffusion models:

https://www.youtube.com/watch?v=iv-5mZ_9CPY&pp=ygUVZGlmZnVza...

I’d like to see more of these models.

I’ve been using diffusion Gemma and it is very fast on GPUs in output token/sec.

In the diffusion Gemma whitepaper, they say they could have done better with more time and compute.

Even with those caveats, it is very uses-able as a local model.

good stuff, no mention of confidence tho, recommend having a look at diffusiongemma and others.