There is no server connection. Everything happens privately on your computer. In order to achieve fast latency locally, we use embeddings or a small fine-tuned model. Code is here: github.com/danielgross/te…
The data is from Kaggle's quotes database, and the embeddings were computed using SentenceTransformer, which then runs locally on ASR. I also finetuned a small T5 model that sorta works (but goes crazy a lot).
Stepping back, I think there is too much GPT-3 "one-shot" in the market now and too little hacking in depth. This demo repo is a bit of a joke but hopefully provides an example of a step in a more serious direction.
Lastly, with M2 MacBooks Pros on the horizon and native Pytorch/metal support, on-device LLMs are a really interesting area of exploration... I hope more people have fun with it in the coming months.
If you're having fun making an AI-native company, shoot me a note!