Settings

Theme

Rate limiter for LLMs outperforms exponential backoff

github.com

2 points by sftombu a year ago · 1 comment

Reader

sftombuOP a year ago

rateLLMiter is a Python rate limiter that smoothes out requests to LLM APIs to get faster, more consistent performance. It uses a ticket bucket algorithm rather than the usual exponential backoff.

Keyboard Shortcuts

j
Next item
k
Previous item
o / Enter
Open selected item
?
Show this help
Esc
Close modal / clear selection