Run QWEN3.8 27B on 16gb Nvidia GPUs
github.comI'm not sure if I want to trust yet another loader with secret sauce, there are already a lot of those around.
What I'd really like is a simple utility that, given my system and a model, will tweak llamacpp to run decently (or tell me it can't be done).
I've been using it for the past few days, and it runs really well!
I usually get 7 token/s using llama or lm studio, but this inference recipe runs at a smooth 80 tokens per second.
Genuinely very usable, and fully local!
Which card are you using? I was getting about 40 with an UD q3 quant with MTP (prediction) enabled and llama.cpp compiled for my compute capability, but was very limited in the context size. I have an 4060 ti 16GB. Wouldn't recommend it as there's a tradeoff between larger context without MTP and about 18 tokens/s.
I got a RTX5080 as primary and a RTX3050 (8gb) as secondary.
I always max out on context size so that means i need to split the model between both gpus + offload some into system ram when using LM studio. I'm guessing that is what accounts for most of the slowdown when i use that.
With this recipe i don't have to worry about it. It's pretty fast, and since its all running in the 5080 with max context it's stable (though i got to look out not to load anything big on the same gpu)
With llama.cpp (CUDA) and a 5060ti (16GB) I get 60t/s with 128K token space. Odd you got 7t/s, did you verify all the model was loaded in VRAM? (--gpu-layers all)
it was but i max out on context so it doesn't all fit with kv cache etc...
7 t/s not sure how you can use this? And if it start thinking a bit. It works yes but that's really very slow. On top it's very low quant that impact the model quality.
> 7 t/s not sure how you can use this
80 t/s - maybe you need to finish reading a sentence before you comment on it?