Ask HN: Local AI product situation
Right now it seems every company wants you to buy their fancy new expensive computer to run local AI models. Why would we want to buy a whole new computer to run local AI models? Why not create a device which has its own memory and chip designed specifically to run models and it plugs into your existing computer thunderbolt port (or event USB 3.2 as loading the model is one time process).
They do... That device which has its own memory and chip is called a GPU...
GPU do other things as well and that makes them costly.
GPUs have instructions to do other operations, memory copy, load custom kernels, but at their heart they are vector (and tensor, many matrix ops) processors. This is identical to what games require, don't just take my word for it :)
The minimal extra hardware functionality is in the output controllers, i.e. the things that control display port, hdmi etc.
Dedicated AI compute processors (e.g. Nvidia Tesla series) are simply GPUs without outputs, and have no built in cooling fans because they sit in server racks that have high powered cooling mechanisms.
There's no extra magical AI specific functionality in them, they just do vector maths really really well because they are designed for it. :)