Show HN: Realtime LLM Chat on an 8GB Nvidia GPU

1 points by z991 3 years ago · 0 comments · 1 min read

Demo runs on a laptop 3070 Ti / 8GB. GPU memory doesn't go above 6GB, so it might run on an even smaller GPU. Uses a 4-bit 7bn parameter alpaca_lora model and performance is significantly worse than ChatGPT as you'd expect.

No comments yet.