What would an LLM OS look like? (2023)
campedersen.com> The future will be agentic.
Minor pet-peeve, but I can't take anyone who uses the term "agentic" seriously. Computer Science has long used the concept of an "agent" and the adjective form of such systems has been "agent based" [0].
There is nothing that makes "agent based" LLMs different than other "agent" systems in computer science, so using the term "agentic", for me at least, is just short hand for "I'm not familiar with the history of AI in computer science".
I don't know if Karpathy has used that exact phrase, and if he has, he obviously is an expert in both AI in particular and computer science in general. But the majority of people I've heard use the phrase "agentic" are just trying to sound cool/cutting edge when referring to a concept that has a long history in the field.
I think "OS" has too much baggage as a term for what Karpathy was describing.
As a term, it implies a specific kind of implementation that doesn't necessarily reflect the high-level properties he laid out - all the user journeys and applications he lists are something you could implement in userspace. They just require interfacing with files and the internet and other software, not kernel-level access to those resources.
Incidentally, I'm working on building such a system. And I have a different thesis: LLMs should be given nice interfaces and APIs to use for the same reason we prefer them. A deep interface, such as the way Linux unifies all the different I/O and networking semantics with shared APIs, makes using the system easier with less risk of mistakes or need to reinvent the wheel. Exposing those to LLMs allows them to spend more cycles fulfilling the user's need as-stated, and less cycles fiddling with hardware and core OS functionality.
Yes, the term has too much baggage. The way to know for sure is that people seem to think he was literally going to make an LLM OS or something. To be fair to Karpathy, I doubt he imagined people could misinterpret what he was saying so badly.
Probably like the oses we have now. Let's see what they say.
E: They want to put LLMs in the kernel, hahaha
EE: Another comment, now that I've stopped laughing, they mention that Moore's law will continue to pay dividends, but we are literally running up to the physical limits of computation already. We've been able to keep things going by going parallel but we're getting closer to situations where the size of the atoms forming the transistors are the limiting factor. That and the heat.
Furthermore, I'm not sure it's prudent to be inventing very energy intensive computing paradigms at a time when the planet is getting warmer and warmer. Doubly so when the utility of that paradigm is dubiously more useful than the "classical" paradigm and costs more than twice the power.
Nice article. We did a demo for booting to LLM and also as Kernel Module: https://github.com/trholding/llama2.c The whole things was funny and buggy, but since then we have been developing in stealth, even trying to raise VC capital. Our goal is to make computers like a buddy to whom you can talk to and explain things and get work done, kinda like a Jarvis. The way we interact with computers haven't changed for decades, its time to disrupt that to get more productivity. I also believe with this approach one can avoid installing different applications, when the computer (models) emulate activities done through applications. For example, cutting and pasting a dog from a dog photo onto a banner for a dog racing competition would not require you to be a graphics artist nor use tools like photshop / gimp. You could tell the computer and it would use segment anything to cut the dog, use Text and SD for banner text and bg paste the dog, seek your approval, search for the fastest, best and cheapest banner printing service and submit it. 10 years ago this could have been sci-fi, but now it is a possibility. Just need to connect the dots, package and polish it to make it a good product.
I yearn for the day I can address my computer as "computer" and have it reply back to me in a dry monotone Majelle Barrett voice while not sounding fake and cheary.
Very similar to what I've been working on for a bit - https://github.com/AmberSahdev/Open-Interface
I think the idea of self-operating computers should get more traction than it currently does. Imagine wanting to create a song and not having to watch 10 hours of Garage Band Youtube tutorials.
Lofty diagrams aside, in practical use it's going to be a minimal hardware abstraction that serves to communicate data between models and allocate compute time based on their importance. Maybe that resource scheduler will be AI but at the end of the day, it's code running on a kernel running on CPUs.
The analogy to an OS was supposed to be an analogy. No one who understands LLMs or OSs actually thinks there should be an "LLM OS" or anything of that nature.
Isn’t this really describing an LLM shell, rather than a “kernel”
Maybe just `import subprocess` and doing some sandbox thing (to not to make any SPOF on host) rather than create a abstraction layer between OS and LLM.
It was meant to be an analogy...