Show HN: Sturnus, a latency-based LLM load balancer
github.comI built sturnus because I wanted the latency-based load balancing of an AI gateway but without a big framework selling an enterprise plan.
It’s a single static binary that does one thing, aiming to avoid the large surface area and security issues that have plagued some LLM gateways.
Traffic is distributed by LLM provider latency and availability. The share going to other providers acts as a probe, so that a recovered provider wins back traffic automatically.
I’ve been running this in prod for a while, feedback, opinions and contributions welcome!
No comments yet.