Explorative Modeling – A new axis for pre-training
arxiv.orgAn exciting new paradigm for generative modeling that acts as a third pretraining axis when added to existing generative models, and also enables end-to-end generation. Increasing exploration monotonically improves existing models across images, video, and language, and the gains grow with scale (7%→36% with data, 13%→23% with parameters). Concretely, Explorative Models (XMs) reach 6.2× sample efficiency, 4.1× FLOP efficiency, and 47% better parameter efficiency. Exploration also enables scaling generalization, and scaling how end-to-end existing models are. As end-to-end generative models, XMs match diffusion on control tasks with up to 256× less inference compute.
I am the first author, feel free to ask questions!
…does K also have to scale? I’d care a lot about wall-clock and peak memory, especially for local/on-device inference.
you can scale it based on the number of modes in your distribution, and keep it low when its not a big issue (e.g. 2/3 worked pretty well). also, this doesn't affect inference at all so on device can still be fast.
Why no qualitative examples? Based on the numbers, you should have some pretty pictures you could put in the paper.