DiffusionGemma and the return of diffusion models for language
Google's DiffusionGemma refines a whole block of text in parallel instead of predicting one token at a time - strong for interactive editing, weaker for deep reasoning.
Although diffusion models show great success in generative image models, it’s quite a contradictory topic for language modelling. This idea comes and goes from time to time, but until now, there wasn’t a stable model that really stuck.
A couple of weeks ago, Google released DiffusionGemma, a diffusion model for language generation. It’s a complete rethinking of how text is generated, opposed to mainstream autoregressive LLMs that work one token at a time, left to right, typewriter style.
Instead of predicting the next token, DiffusionGemma starts with a noisy 256-token canvas and refines the whole block in parallel, bidirectional context, revisions across the block. Less typing. More editing a draft.
It fits interactive work like code infill, real-time editing, and fast drafts especially on local GPUs where latency matters. It struggles with deep reasoning, math, and complex instructions, where autoregressive models are still stronger.