Repligraphs: A Primitive for AI Provenance
lukechampine.com
1 thread
How does this work since LLM outputs aren't deterministic? Most users don't have full control of their LLMs.
True, you can't do this with GPT or Claude, at least not today. But you can do it with a local model: vLLM supports deterministic inference out of the box! (If you have a MacBook, you can confirm this yourself; just point your agent of choice at the site. It will download a small open model and run it with vLLM Metal.)
Naturally, I hope that OpenAI and Anthropic will one day offer deterministic inference; see the "Weights" page for what this might look like.