Will it be possible to do LLM Pool Training?
Inspired by Crypto Pool Mining: would it be possible to train LLMs as a pool of volunteers? I feel like that would be a good way to have frontier models that are open and don’t rely on big corporations with data centers to train and release them…
2 threads
Datacenter-scale GPGPU compute is extremely efficient compared to what you would achieve with a network of independently operated distributed heterogeneous devices spread across the world. Just imagine the overhead for ensuring that a node doesn't poison the gradients. And that's just one very small part of the problem.
possible, yes, people already have projects
practical, not yet, if ever (?)
the difference is data and calculation dependencies, in crypto they are independent