Settings

Theme

Running a vLLM LXC on Proxmox 9 with Nvidia GPU Passthrough

medium.com

2 points by jakeasmith 2 months ago · 1 comment

Reader

jakeasmithOP 2 months ago

I've been running Ollama in my home lab for awhile now but I wanted to experiment with running something a little more "low level". I saw a ton of posts about Llama.cpp, which looked interesting, but there wasn't a lot specifically about running vLLM on Proxmox, so I thought I'd give it a try. Setting up vLLM in an LXC wasn’t necessarily difficult but, even after doing it several times, it was still tedious. These are the notes I’ve taken along the way, if only for my own reference next weekend.

Feel free to tell me what I got wrong :)

Keyboard Shortcuts

j
Next item
k
Previous item
o / Enter
Open selected item
?
Show this help
Esc
Close modal / clear selection