TextSynth Server
bellard.orgI've been feeling FOMO (for lack of a better term) about recent AI & ML/GPT progression.
It feels like ML/AI it might be the beginning of the end for a large class of things (if I wanted to be alarmist I'd say "everything") -- and the fact that Fabrice Bellard has jumped in and done the absolutely obvious rising-tide thing (building an API that abstracts the technologies) speaks volumes.
Releasing something like this fits to Fabrice's pattern of work -- he built Qemu and that served as a similar enabling fabric for people to run virtual machines. QuickJS quietly powers some JS-on-another-platform functionality.
Simon was right. The Stable Diffusion moment[0] is already here. It's going to accelerate. It was already moving at a speed that was hard to follow, and it's about to get even faster.
There are too many world-changing things moving forward at the same time, and I'm only looking at such a small cut of the tech sphere. I don't know what to do with myself, I feel so thoroughly unprepared.
Every one of these articles fills me with a related kind of dread.
My whole life, my whole personality is architected around making things by hand for other people. My ideal world is a hipster stereotype where we all sit around using a small number of artisanal products to make other artisanal products for each other.
The arc of my programming career has gone lower and lower down the stack because when I create, I enjoy it most when it feels concrete, deliberate, and long-lasting. I get no joy out of duct taping a few libraries together (though I respect others who do).
While I spend a lot of my day doing code review and think it's a valuable, important part of the process, it's not my favorite task. I like making stuff, not just socially interacting with others to loosely guide them towards making stuff. The idea of AI-assisted software development to me just sounds like taking the one part of the job I like most—writing code—and turning it into even more code review, except now I'm reviewing code vomited out by a machine.
And I completely dread the long term societal implications of a world where most people spend most of their day consuming media auto-generated by a machine. Where lonely men and women hide from their social anxiety by cultivating simulated romantic relationships with chatbots. Where teens have their expectations of sex set by watching synthesized porn starring virtual actors doing things that are physically impossible. Where people watch auto-generated videos of impossibly idyllic vistas instead of actually leaving the house and going for a hike. Where our beliefs of the world are formed largely by synthesized news articles that may or may not accurately reflect it. Where children learn to speak, read, and write from AI tutors and pick up all the grammatical and stylistic quircks of the AI model such that they now because actual real parts of human language.
And, of course, where almost all of the massive profit generated by all of that flows to an increasingly small number of huge corporations.
None of that sounds like a world I want to live in.
I totally get the value of AI for things like classification and understanding. But generative AI feels like a pandora's box to me.
Interested to know what your plans are. You've thought about this a lot, and for me your books sit next to all the others I have on C, assembly, math, etc--and, I am a bit embarrased to say, a large portion of which I still need to study.
I don't think I should despair too much, and simply grant triumph to a statistical steamroller.
The value is we'll still be making stuff, or yearning for it, and AI becomes a part of our toolkit (or never). Perhaps in your case, we will see Prompt Engineering Patterns someday.
I intend to plop down a tiny fortune enabling my children to have obscene hardware for wherever their personal projects take them, and server-grade CPUs, multiple GPUs, and fiber will be a given.
Just know you're standing at a height somewhere above, and as a giant to me I hope you can see farther ahead :)
Maybe we'll retreat more into our personally-named server, managing a handful of like-minded users, crafting rooms in a MUD no one will read. We'll publish stats for our packet filter, read stories typed by hand, and make little games.
There are still lots of problems we would be completely new to in many domains, people suffering injustice, and are those not things we might be interested in as well?
For sake of learning things, we are still satisfied. Even if AI were to generate it in an instant. For sake of satisficing our home labs and side projects, we find a tiny reprieve.
> Interested to know what your plans are.
I think I'll manage to eke out the rest of my career doing the kinds of programming I enjoy at least until I'm ready to retire. That's basically the extent of my plan.
> Just know you're standing at a height somewhere above, and as a giant to me I hope you can see farther ahead :)
I wish I could, but I'm as lost as the rest of us. I just happened to have written a couple of books.
> Maybe we'll retreat more into our personally-named server, managing a handful of like-minded users, crafting rooms in a MUD no one will read. We'll publish stats for our packet filter, read stories typed by hand, and make little games.
That hits perhaps just a little too close to home.
I think of the countless old folks in garages tinkering on old cars with manual transmissions and no engine computers, simple enough that they can still be tinkered, wishing they still made cars like that today. (I drive a twenty-year-old truck for similar reasons, though I'm not savvy enough to actually work on it.) Or other old folks hand-knitting sweaters and blankets for grandkids who appreciate the gesture but put it into storage because the latest mass produced fast fashion blanket they got off Amazon matches their room's decor better.
The whole thing just makes me feel old and out of touch.
> My whole life, my whole personality is architected around making things by hand for other people.
Not sure if there's a conflict here. We still use compilers, right? Unless we write in machine instructions, there's always some kind of program generations somewhere.
> I get no joy out of duct taping a few libraries together (though I respect others who do).
You still can work on stuff that requires human ingenuity . If you look at the the credits section of the GPT-4 paper, you will see many people there because they are the master of "low-level" optimization. For instance, the author of Reformer is a lead in the Long Context section, because apparently he knows how to reduce the O(N^2) of self attention to O(Nlog(N)), to say the least.
This is specifically what the situationists were trying to express with “the society of the spectacle”. It is very interesting to live though a time where broad swaths of humanity are awakening to such a critique.
I understand and share your fear. In a sense, I think we already live in this world: there are many people who are already slaves to YouTube, TikTok, strategically timed push notifications, etc.
We live in a world where digital tech is fantastic for people who have the skill and strategy to place healthy limits on their own use of it, and the worst possible world for those who cannot. And it's all about to get much more extreme.
All of your fears will likely come to pass, but I think there are also much more positive framings and there are fantastic opportunities to build solutions to some of these problems.
- Loneliness. Today, lonely men and women already suffer. A simulated relationship is a legitimate improvement over no human contact at all. Could a bot use the embeddings of your interactions to match you with a compatible partner, saving you the anxiety of having to navigate the dating market.
- Auto-generated videos. Frankly, most movies and TV shows suck today, probably because they come out of a broken Hollywood. I can't wait until the barrier is so low that a random genius college student can make a feature film with almost no resources. Maybe there will be more new movies that are actually good!
- AI tutors. Playing with GPT4 the past few days I've already experienced visceral joy from its use as a teaching tool. It is like having a pretty smart co-worker who knows something about literally everything. I'm honestly so excited about being able to learn new things without dealing with the fundamental barrier of finding good sources of knowledge.
> We live in a world where digital tech is fantastic for people who have the skill and strategy to place healthy limits on their own use of it, and the worst possible world for those who cannot.
I have several family members with ADHD and this is profoundly true. I don't know if they would even have a diagnosable condition if they didn't have the misfortune of living in a world that is actively preying on their attention every moment of their life.
> Today, lonely men and women already suffer. A simulated relationship is a legitimate improvement over no human contact at all.
I disagree. What I see over and over again in society today is that parasocial relationships (fandom, celebrity worship, life vlogging, etc.) enable and intensity loneliness. They provide just enough of an emulation of actual human contact with none of the effort or risk that many people (myself included, sometimes, honestly) do that instead of being forced out the door to get the real thing.
It's exactly like the thing we all do where you wander into the kitchen hungry. You're too lazy to prepare a real meal, so you have some chips or crackers or something and that gives you just enough satiety that you wander off again. But then ten minutes later you're back again, still hungry.
> Maybe there will be more new movies that are actually good!
I would so much rather watch a movie with flaws and that isn't entirely to my taste if it was made by a person who actually cared about what they were doing.
> There are too many world-changing things moving forward at the same time, and I'm only looking at such a small cut of the tech sphere. I don't know what to do with myself, I feel so thoroughly unprepared.
I think the general trend is that actual useful applications are emerging from enormous models trained and owned by billion dollar companies only. Even projects that aim to run models on private consumer hardware are dependent on commercial orgs to produce them. It doesn't seem like that is likely to change.
I don't think there are many positions/jobs/roles for people doing integral, foundational work that requires a deep understanding of ML. Becoming a world expert in ML will probably only open up opportunities at a dozen companies.
A very shallow surface-level understanding of ML already puts you leagues ahead of the general population. In terms of job security, figuring out how to use an ML API will get you hired faster than knowing how to advance the field.
We don't actually need that many Fabrice Bellards.
> I think the general trend is that actual useful applications are emerging from enormous models trained and owned by billion dollar companies only.
One way to think about it: Today's LLMs require incredible outlays of capital and processor power (and crews of folks with doctorates), such as billion dollar companies can provide. But how is that different from what Intel brought to commodity CPUs in the '90s/'00s, or what Nvidia brought to GPUs in the '00s/'10s? Or even what Cisco and folks brought to networks?
Though we may never design an artisanal CPU/GPU/router, we get to work with them every day to make things, and to communicate. These LLMs can be that for us at this moment. Let's go out and enjoy them, and see what we can make within their (vast) domain-specific capabilities.
[takes off rose-tinted glasses]
> But how is that different from what Intel brought to commodity CPUs in the '90s/'00s, or what Nvidia brought to GPUs in the '00s/'10s? Or even what Cisco and folks brought to networks?
Yes, they made these technologies accessible and useful. And very few people needed to understand high-K dialetrics or out-of-order execution to use them, hence I think the FOMO is misplaced
Same here. I don't know if I retire or dive in deep. There's no really other way to sound other than alarmist.
This is the biggest revolution I've seen in my lifetime and it's going to make the PC, internet and smartphone a kids toy.
> Same here. I don't know if I retire or dive in deep. There's no really other way to sound other than alarmist.
It might be a blind-leading-the-blind situation, but I wonder if retiring will protect you. Could you imagine having gone into retirement in the early internet? Wouldn't you be really confused at how to use things today? Or maybe you would just learn most useful things passively as a consumer.
It feels like the gap between the people who can make and the people who consume is widening.
> This is the biggest revolution I've seen in my lifetime and it's going to make the PC, internet and smartphone a kids toy.
This coupled with everything else that's happening is all too much. Getting closer and closer to cheap/free energy (batteries, solar, fusion, wind, etc), AI/ML, Robotics...
To me it seems clear that most people are going to become a kind of peasant or serf, effectively "owned" by their techno-corporate silo. Programming as a human activity will resemble professional sports or writing: a handful of well-paid superstars and the rest of us mostly just watch/read (although of course we play sports and write, and basic computer literacy should be a thing, eh?)
This might not be so bad? As you've no doubt noticed, most people suffer from some combination of ignorance and stupidity. We can barely function. An earthquake happens and half the building fall down, despite (some) of us knowing how to build them so they don't do that. That's not a technical problem, it's a social problem, eh?
As far as technical problems go, we have pretty much solved "Life on Earth". In video game terms we beat the boss about a century ago (give or take a few decades) and now we're just sort of running around acting silly while the end credits scroll. Atomic power, materials science, etc. We licked it.
IF we employ these newfangled talking computers to tell us how to solve our problems, they will, and we can live happily ever after. (prompt: "What would Jesus do?") It doesn't matter that they're merely regurgitating the wisdoms of the ages, when the computer says it, maybe it will stick?
I understand what you mean. I have dropped delving into other programming languages and frameworks to learn more about these AI/ML/GPT stuff.
Control freak, by any chance?
Is it really so hard to just "go with the flow"?
sorry a bit late with my response, but original commenter here.
I'm not much of a control freak -- it's more when you see an credible posisble extinction event in the distance, don't you want to act?
And even if you didn't view it that grimly, isn't the ML stuff just SUPER cool? Trainable prediction engines are really amazing and very actually useful (feels like the first iteration was being able to recognize things in images, which felt like magic).
Going with the flow is awesome, but one thing I've found about life is that if you're not in the right flow, it's completely different. It's not like you have to be in the perfect stream, but you need one with fast moving water.
Imagine being 10 years late to computers or the last one in your area to get a typewriter.
I personally feel like I have to immerse myself in stuff to get it, and the lack of more than surface level understanding of ML is worrying with how big it could potentially be.
Two things to note:
No, indeed, if I see an extinction event in the distance future, I dont feel a need to act. It is hard to explain, but I dont feel responsible for human kind. Its a nice experiment, and I love to be part of it at the seemingly right time. But I dont go as far as investing personally regarding the future of humanity. If we make it (to wherever that would be), fine by me. If we dont, thats also totally fine...
Regarding the part that it is hard to keep up-to-date with current ML: thats true of course. However, what helps me here is that I feel like this is interesting, but not my field of work. Tinkering with free software was far more contagious, because I could do that literally at home, without wasting too much money. ML is totally different. I could toy with tinyGPT, but that is effectively too tiny to be fun. Once I scale up, I no longer can do stuff on my own. Sure, I could also order 1024 A100s from a cloud provider, but... that is going to bust my budget. So in a sense, this makes me free from wanting to understand it all. Its totally OK for me to have a rough overview on how it works...
I have a positive outlook for large language models where we can have new synthesized stories that break the rules of what we consider what makes something a game or not. The biggest complement to new forms of art are the people that reject the new forms.
I can’t really sing or draw anime but now my computer can so it can be a large enabler for people.
Hard to just "go with the flow" when your livelihood can be possibly threatened.
Well I’m sure that one guy will be able to change the course of history to suit his job situation. It’s going to happen so you had better go with the flow. Become water.
In the US, people's jobs effectively justify their existence. Smugly saying mid or late career professionals can "become water" when their entire career category faces collapse is the most patronizing thing I've heard in a long time. That's a non-recoverable blow to many smart, capable people. There's a big difference between demanding we stop the wheels of progress to protect a few people and saying we owe the masses being crushed by them some harm reduction. Of course society on a whole will profit. But, the rust belt shows that the platitude about people deemed professionally unnecessary just "figuring it out" merely comforts the people turning those wheels.
Not all that long ago people on this very site were super enthusiastic about the prospect that my job would very likely be taken over by the robots.
Because, you see, the robots didn’t need to be perfect but merely fractionally safer than the humans.
Now all I see is No True Scotsman arguments on why the current crop of AI won’t take away their jobs and, ironically, my job is pretty safe due to Tesla’s tomfoolery causing blowback from the feds.
So robots eating the world is good or what?
Yeah, it's pretty bizarre. So many people who become incensed by businesses profiting from being shitty to customers are gleefully just dumping the clutch without looking for people in the crosswalk. Maybe literally if the car is driven by an algorithm. I'll bet they could whip up a lot of macho, patronizing personal-responsibility-based arguments in support of being able to mow down the slower pedestrians at crosswalks though.
Remember how the universe works? It doesnt care about the well-being of individuals. Thats just a fact of life. Demanding equality of outcome was always an illusion, doubly so in the face of big changes.
Demanding equal outcome is very different from saying we shouldn't kick the chair out from under someone and say it's their own fault. Homo-sapiens didn't become the dominant species on the planet because people were stronger, more aggressive, or greedier individuals. They thing that made them better was cooperation.
So that's a fact of life? Families don't work like that. Military organizations, governments, and and sports teams don't work like that. Businesses don't work like that internally. Well, not successful ones. Sears actually tried that and promptly collapsed within a decade-and-a-half of pointless infighting and undermining. The fact that you think that's a natural state of the economy and not a deliberate choice proves how great the propaganda has always been for lazzais faire American capitalism.
If this is a "non-recoverable blow" then those people were likely not as smart as claimed to begin with...
What a ridiculous thing to say. People's brains become less plastic over time and the longer we stay in one specialty, the less we keep up with unrelated tech and knowledge. Intelligence had no bearing on whether a late career radiologist could pivot into a totally different category of job if deep learning eliminated the need for the only medical career they were qualified to do. As a long time developer, it blows my mind how common this hubris was among my colleagues. As depressing as it will be to see the overwhelming majority of software developers rendered obsolete by progressively more sophisticated code generation tools, I will be glad to see the naive arrogance in the software world taken down a few hundred notches. It's an industry full of people convinced that they are too smart and useful to get left behind. Lol good luck.
> All is included in a single binary. Very few external dependencies (Python is not needed) so installation is easy on most Linux distributions.
I have to disagree. The combination of being closed-source and dynamically linked makes a program a hassle to run on Linux. Even if it isn't at the moment of release, it soon becomes one. While ts_server is better than most, it already requires an old version of libjpeg-turbo not available in my distribution's repositories. I had to run it in a Rocky Linux container:
docker run \
--rm \
--mount type=bind,source="$(pwd)",target=/app/ \
--publish 127.0.0.1:8080:8080 \
rockylinux:9 \
sh -c 'dnf install -y libjpeg libmicrohttpd && cd /app/ && ./ts_server ts_server.cfg'
The ABI doesn't seem to be the problem, and Win32 does not include a libjpeg, so your Win32 approach would only work if it also bundles or statically links libjpeg.
> your Win32 approach would only work if it also bundles or statically links libjpeg.
Of course. The stable ABI is to allow your bundled DLLs to keep functioning. (Check out https://news.ycombinator.com/item?id=32471624 for an extensive discussion of the link.)
The Fabrice Bellard web page must be one of the most underselling ones on the entire web. So many amazing projects. Not a word that really emphasize the importance, coolness. Just a simple list with short factual descriptions.
These kind of people are like that. Does Linus Torvalds have a web page? No. He knows he doesn't need one. They are gods in the IT industry, and they know it.
He has a blog because I recall reading about him reverse engineering a file format for his wife’s embroidery machine.
Very interesting as usual from Fabrice Bellard, but I'm a little bit disappointed this time, because libnc is a closed source DLL. Nevertheless it will be interesting to compare it to the amazing work of Georgi Gerganov: GGML Tensor Library. Both are heavily optimized, supports AVX intrinsics and are plain C/C++ implementation without dependencies.
I expect LibNC will be better in every aspect: performance, accuracy, determinism. But hopefully with time we will close the gap.
The comparison table from the ts_server site looks awesome though. I wish we could generate one for llama.cpp, unfortunately too busy with other things at the moment.
Hey liuliu, would love if you join the project when you find the time - your work is really inspiring!
Thank you for the compliment! Definitely would love to contribute in some ways, once we have the benchmark, I have some ideas :)
refreshingly humble take -- thanks for your hard work. The work you've done and put out in the open is massive.
ChatGPT is pretty good at disassembling x86, and is able to give reasonable descriptions of what the code is doing (e.g try "disassemble the following bytes in hex and explain what they appear to be doing: [bytes from a binary in hex]")
I'm curious how soon someone uses these models to effectively ruin the ability to use releasing binaries as an obfuscation method.
Using ChatGPT as a disassembler seems like a dumb idea when free disassemblers already exist. What possible advantage does it give?
The ability to explain the code, and extract higher level understanding. Disassembling into raw instructions is the most trivial part of reverse engineering an application. Hence "and explain what they appear to be doing" bit.
For the pieces I've tested, it often recognises the source language, and could give ideas about what the code was for and what it did.
> Disassembling into raw instructions is the most trivial part
So why not do it in the proven-correct tools and give ChatGPT the instructions?
I'm all for finding neat use cases but I wouldn't use an AI chatbot as a calculator...
You could do that too, but that is entirely missing the point, which is that ChatGPT is capable of inferring higher level semantics from the instructions and explain what the code is doing. You're getting hung up on a minor, unimportant detail.
Apparently the point is proving it's possible. Not making it useful.
No, that was not the point at all.
The point is that ChatGPT understands the code well enough to explain what it does, and so there's reason to wonder how soon someone leverages that in a disassembler to produce far better output to the point where using releasing "only" the binary as an obfuscation mechanism stops being viable.
E.g. additional value would be comments about purpose, labels for function entry points that makes sense for what the function does, labels for data that makes sense for what is stored there, comments with explanations of why things are structured the way they are.
Having reverse engineered programs from binary in the past, inferring good names and commenting the sources is the vast majority of the effort.
Thanks. That's at least the start of the type of thing I was thinking of.
>E.g. additional value would be comments about purpose, labels for function entry points that makes sense for what the function does, labels for data that makes sense for what is stored there, comments with explanations of why things are structured the way they are.
None of this requires giving it a binary. You are asking it to do 2 tasks, both of which it will do with some level of error. You could disassemble the binary near perfectly, for free. You have a hammer and everything looks like a nail.
It was an example. I've already pointed out to you you've gotten hung up on a minor, unimportant detail of an example. You're right, you can do the raw disassembly of the instructions by different means first. Which is entirely irrelevant to the point made. In other words: You're being incredibly obtuse and arguing against an imaginary point nobody made.
The recommendation was literally to try the prompt. The purpose of that recommendation was that if you tried it you might see some of the additional commentary ChatGPT adds, which is where the value was. I could have suggested you disassemble it first, but it seemed easier cut and paste the hex values. That is all.
Do you have anything you want to discuss about the actual point? If not we're done here.
It will struggle with understanding small parts of big programs without seeing the full context, though maybe you could get around that by making it generate some sort of summary for itself or something like that
My tests on small parts of big programs suggests if anything that it does far better than I expected, but you're probably right that it would struggle with that for many things if you tried turning it into a bigger tool, and having it generate summaries is probably essential. While we can "fake" some level of memory that way, I really would like to see how far LLM's can go if we give them a more flexible form of memory...
There are also equally easy to use open source systems, check out this one for example: https://github.com/kuleshov/minillm
Anyone know what format the models have to be in for use with textsynth? I looked at the gpt2 example binary (gpt2_117M.bin) and it seems like the "normal" params.json is embedded as a header for the binary and then some ascii string like "attn/c_attn/" and then the binary weights.
I tried just using the Stanford Alpaca fine-tuned version of the llama 7B weights that work with llama.cpp with textsynth but it didn't like that (ggml-alpaca-7b-q4.bin: invalid file header). Having a textsynth HTTP API would save me a lot of hassle . I'm currently wrapping the stdin/out of a execution of a modified llama.cpp binary and that's extremely messy.
Question for you (sorry I don't have an answer for you): Where are you storing the models relative to the ts_server folder?
In comments on this post, and elsewhere on other posts about AI, I see a lot of people referring to worries around the potential for lots of types of jobs to be heavily impacted by this technology.
I feel like people are often referring to 'coding' when they express these worries. You know, actually writing code, having been given a spec to do so, and perhaps also participating in code review, writing tests, all the usual engineer stuff.
My question is, amongst the HN crowd, what kinds of roles or areas do we think might be somewhat immune to this effect? The first thing that occurs to me are security, infrastructure & ops, networking. And of course the requirements gathering stage of software development. It is already the case that a lot of senior devs probably don't write much code and spend more time on communication between different stakeholders and overseeing whoever (or whatever) is writing the code.
Anyone else been thinking about this? What tech roles might thrive in the face of AI.
In my day job I work with TypeScript and React, and if I'm honest, I'm not worried at all. Why should I be? Automation is all around us, and has been for years.
The way I personally see it, is that AI such as ChatGPT is another tool in our arsenal that we as developers will have to figure out how it fits into our workflow. I think long term it will help us write better code, and in general be more productive. For example, less time trying to find answers hidden deep in Stack Overflow as we'll be able to get that information directly from ChatGPT.
I can completely see that some smaller places they might not require a developer and instead use ChatGPT to write code, but it still has to be verified and all the other processes around making that code "live", etc.
If anything I'd be more worried if I were a copywriter, as I think it's an under appreciated skill and companies may think they can get away with ChatGPT and a quick glance over the copy.
Either way, I'm positive and look for new ways to help me come a more productive and well-rounded programmer.
I'm not at all worried, either. Because I know we already went through this. Companies tried to outsource their work to India, the Philippines, and elsewhere, and it ended in tears and expensive reversals of strategy. They discovered that someone has to check all the work.
No one wants to pay for code reviews. They don't show up on the agile board and we all collectively pretend they are "freebies" the devs give out to the company. Just like unit tests and all the other shit that we just expect devs to do in their spare time between 3 hour meetings and feature work.
I can see ChatGPT being an augmentation. But that only works because you have a human dev that does the merging and can take the blame if their code goes wrong. Remove that human and you're staring into the abyss.
I agree with this take, in the main. I think the level of FUD elsewhere is probably unwarranted. Though I fully subscribe to the idea that it is very disruptive tech.
In the context of software engineering I am tending to see it as another layer of abstraction. Once upon a time there was perhaps not much above machine code / assembly, but now you can have quite a few layers providing abstractions over that ending up perhaps at Javascript, or maybe low-code tools. For me, AI sits somewhere vaguely in that category (though with a much higher level of sophistication).
> what kinds of roles or areas do we think might be somewhat immune to this effect?
On what time-frame?
Permanently immune? You have to postulate that there's something a human can do that a computer system just can't. To me, given the progress we have already seen, I see no strong reason to imagine that there is such a thing. More precisely, it becomes a metaphysical question about what it means to be human, "What are people for?".
- - - -
In the medium term (and this may only be a few years) the role that will come to the fore is that of the human-computer psychologist, so to speak. We already talk about "prompt engineering". There are two questions: goal and context. What do you want the computer to do, and how do you know when it's doing it successfully? -and- What are the side-effects, the "ecology", of the selected solutions? Especially, otherwise unforeseen side-effects.
This man, Fabrice Bellard again...
Frankly, I have not seen a more impressive portfolio of programming output.
I wonder if he ever went to interview they would try leetcode, or what is that squiggly thing in a code sample is for or are you a Language XXX programmer and other similar things. I can bet that this guy would probably fail many standard selection criteria.
I can bet that he wouldn’t.
For me, the most interesting part is the statistics on all the models. These show that 8 bit quantization is basically as good as the full model and 4 bit is very close. This is the first time I see such table across a large number of models in one place.
Pretty much.
Llama specific:
https://github.com/qwopqwop200/GPTQ-for-LLaMa
> According to GPTQ paper, As the size of the model increases, the difference in performance between FP16 and GPTQ decreases.
https://nolanoorg.substack.com/p/int-4-llama-is-not-enough-i...
https://docs.google.com/document/d/1wZ0g9rHI-6s7ctNlykuK4W5T...
Expect to get away with a factor of 4-5 reduction in memory usage for a minimal loss of quality. :)
"The CPU version is released as binary code under the MIT license"
This gives off the surreal sci-fi vibe that the binary is the source. And who knows... true wizards work in mysterious ways.
Very inspiring stuff! I hope one day we get to see the magic behind libnc.
Is it just me, or is the llama model not available to download there?
Edit: nevermind, the models are all there, just some of the links aren't.
If there were Oscars or Nobels for programming, Fabrice Bellard should have won one long ago!
I guess the ACM A. M. Turing Award is as close as we get, which AFAIK, they never won. But Bellard won countless of other awards, competitions and benchmarks, it is not like they are unrecognized for their work in the field.
Is it necessary to pre-process models from hugging face before using them with libnc ?
Looks like he previously provided a conversion script for his discontinued gpt2tc.
I disagree with the linked post, most people use 'REST' to refer to JSON-over-HTTP now.
You disagree with the guy who created the term REST?
Yep.
'Most people' can call it what they want, but in this context, they're not referring to a RESTful API. Search for 'REST JSON API' and you may discover 'most people' may be a subset of 'some people', and not a majority of people.
> The GPU version is commercial software. Please contact...
Shame.
The GPU version of libNC is available as a free binary, and you can find MIT-licensed source code implementing (training and inference of) transformers using libNC at https://bellard.org/nncp. (nncp was meant to be a submission to the Hutter Prize, which would have required open-sourcing libNC too, but it didn't qualify due to using AVX2 and too much RAM. At least the CPU version binary is MIT licensed since ts_server is.) I think it wouldn't be that big a project to support LLaMa starting from that code, although it is dense code.
Edit: licensing
That version of libnc_cuda.so doesn't seem to be compatible with ts_server though.
I think it's fine if a software legend (or anyone) wants to make some income.
"Shame. I cannot take the code / binary and run it as a SaaS efficiently." - HNers
Good for him to commercialize it, and at least he is not pretending to be a non-profit accepting VC money.
Countering SaaS-ifying by others can also be achieved through the AGPL or through the BSL (initially not open-source, reverts to open source after set period).
I do believe one of the failures of GPL was not being AGPL from the start.
Shame as in "it's a shame" or as in "shame on them"?
The former. Makes sense for their business model.
I'm pretty sure a dev skilled in ML and GPUs has no trouble modifying the MIT code (which runs on the CPU) to something that runs on GPUs...
The CPU version is provided under binary format only.
You're right : `The CPU version is released as binary code under the MIT license`. That's quite an unusual choice, not even sure how MIT would apply to that...
You can release whatever you want under MIT :) It grants you the right to use the binary for commercial projects (or any type of projects you want), to modify the binary, distribute it yourself and more. You cannot hold Bellard/the license holder liable for anything related to it, and you must include the license and copyright if you distribute it.
Seems pretty doable to me :)
I guess you could decompile it and redistribute that source if you were dedicated enough
The goal here is to allow you to redistribute while maintaining the copyright notice.