Ask HN: Why can't Nvidia compete in the inference market?
Hey HN!
Lately I have been thinking about how the inference market will involve. We saw that llama 3 with groq was showing great results. Once question I had was why can't nvidia compete in the inference market? In the video below chamath was saying how nvida won't be able to compete in inference.
https://youtu.be/1ZQ33OnGFWE?t=1254
is here just chilling his bag or are there some truth to this? What are fundamental reason that this would be the case? Likely because they don't make inference accelerators, they make GPUs. It's entirely plausible that someone could optimize the hardware pipeline better by incorporating less complex logic; I wouldn't expect Nvidia to try that though, not with their main products. This reminds me of a sentence Jensen said internally: "We do the thing that only we can do"