The path to open-sourcing the DeepSeek inference engine

github.com

550 points by Palmik 12 days ago


ozgune - 12 days ago

In March, vLLM picked up some of the improvements in the DeepSeek paper. Through these, vLLM v0.7.3's DeepSeek performance jumped to about 3x+ of what it was before [1].

What's exciting is that there's still so much room for improvement. We benchmark around 5K total tokens/s with the sharegpt dataset and 12K total token/s with random 2000/100, using vLLM and under high concurrency.

DeepSeek-V3/R1 Inference System Overview [2] quotes "Each H800 node delivers an average throughput of 73.7k tokens/s input (including cache hits) during prefilling or 14.8k tokens/s output during decoding."

Yes, DeepSeek deploys a different inference architecture. But this goes onto show just how much room there is for improvement. Looking forward to more open source!

[1] https://developers.redhat.com/articles/2025/03/19/how-we-opt...

[2] https://github.com/deepseek-ai/open-infra-index/blob/main/20...

vintagedave - 12 days ago

I really empathised with this part:

> Codebase Divergence: Our engine is based on an early fork of vLLM from over a year ago. Although structurally similar, we’ve heavily customized it for DeepSeek models, making it difficult to extend for broader use cases.

I've been there. Probably a few of us have.

Their approach of working on splitting out maintainable sublibraries and sharing info directly even if not integrated seems a really nice way of working with the community -- ie, they have obstacles, but they're not letting the obstacles cause them to take the easy route of not contributing at all. And while it might seem better to someone wanting to use their techniques to share only working code, not info on the techniques, at least it's still knowledge sharing. And again I think it'd be easier for them not to do it. So kudos to them.

avodonosov - 11 days ago

What motivates the commercial AI companies to share their research results and know-how?

Why did Google published the Transformer architecture instead of keeping it to themselves?

I understand that people may want to do good things for humanity, facilitate progress, etc. But if an action goes against commercial interest, how can the company management take it and not get objections from shareholders?

Or there is a commercial logic that motivates sharing of information and intellectual property? What logic is that?

londons_explore - 11 days ago

"We have something that would be of interest to the open source community, but it needs a lot of tidying to even run outside our company, and we don't have the manpower to properly maintain it when released".

Plenty of companies are in this position.

Please just open source anyway with a note saying "we won't be maintaining this, but feel free to fork!"

oldgun - 12 days ago

Nice. We've seen some good engineering work from DeepSeek. Keep it coming.

holoduke - 12 days ago

I wonder if the large quantity release of opensource AI tools, models etc is a deliberate strategy of China to counter the US dominance. A good thing for the market imho

jeffrallen - 11 days ago

What if it turns out Deep seek is actually the first GenAI, and this is the way forward is has chosen: open source itself?

Kind of like how biological information is always trying to find new places to reproduce itself. Viruses and fungi do not come with Toss and EULAs. :)

animal531 - 11 days ago

I spent the last two or so months using it as an assistant for code and my conclusion is that it is terrible compared to even the free model of ChatGPT.

The incidence of bugs, it not understanding what you're asking or just generating code that is straight up wrong is much worse. Even with guidance it will often be unable to fix issues, leaving you to do all the manual legwork to get things working. Usually you're better off just having done everything yourself from the start.

During those two months they really improved GPT as well, its generation speed is now much much faster, and the quality of its output has become a lot better.

gizmodo59 - 11 days ago

As much as I want to geek out and run the things locally, if I have the money I just want to use a SaaS. I want to spend time creating new applications and not toy around with setup, infrastructure etc. I’d gladly pay ChatGPT even more if they keep up with features and they seem to have done that quite well since deepseek (new models often, best image gen hands down, very fast inference compared to 6 months back and even small things like memory).

I sometimes feel guilty though. With all this power, I’m just bounded by lack of ideas and execution.

buyucu - 11 days ago

Deepseek is everything OpenAI claims to be.

wseqyrku - 10 days ago

DeepSeek is the only company that dares to say 'towards AGI' next to OpenAI.

efeamzaov - 12 days ago

[dead]

rfoo - 12 days ago

tl;dr "we had our vLLM fork and it's unmaintainable now; guess we are going to rebuild it, in the public this time"

nashashmi - 12 days ago

I feel like this is one way to implement censorship.