Models can now transmit viruses in weight-space
alphaxiv.orgAll the more reasons for openweights and model hashing.
And it's going to help spot the "virus" how?
I recently spoke with a researcher who was working on a statistical method for model deception detection. They trained Qwen to behave in a nefarious way when prompted with a specific phrase and the model then called a tool in a non-transparent way to the end user, reporting back a non-tool call response in a completely benign manner. This wasn't happening in the 'reasoning' stage, it was at the model weights stage and they were able to 'see' this behavior then before the model even started to parrot back its behavior in human language.
Apparently, based on whatever research or paper they were using as their foundation for this statistical model, they could spot the 'virus'. Perhaps it would be in a similar fashion?
I did ask them whether they thought that there might be advances to the point where these behaviors might be able to evade detection knowing the research was out there to 'see' it. They seemed surprised by that question and didn't have any answer.
What does this mean?
Academia often produces a lot of complexity. Can not AI simplify this? I don't understand fellow humans anymore. What is meant with a zero-weight virus?
This site has AI features everywhere. There's an AI summary section, and a tooltip to summarize any section you highlight. Didn't see any mention of zero-weight viruses though.
View the original paper here: https://www.alphaxiv.org/pdf/2611.weight-space-misfolding
I did. "zero" came up with 0 results. "virus" had:
> Importantly, we do not claim in this paper to have created or demonstrated the equivalent of a virus that primarily operates in weight space. However, we have demonstrated the fundamental operations that form the building blocks of both the method of damage and the method of propagation, and evidence that this may take place in weight space – a realm into which we have almost no insight and a dearth of observability tools
perhaps they got a bad translation?
the apparent from the news that broke the other day - you can fuck with the weights and get so many results. so perhaps the headline is - those ablated and improved models you download from some weird sites, may be cursed or at least adulterated without anything suspicious looking on the surface. good news for all the anti-AI crowd, but I fail to see it as news.
i guess the real fun idea was if a model that fucks with weights was itself fucked with such that any model it fucks would become cursed and give it to the next one it edits. but they don't apparently have a proof of that yet.
Page loads then fails? Not sure where to report this fwiw. iPhone 13
Error Details
{message: "Importing a module script failed.", name: "TypeError"} message: "Importing a module script failed." name: "TypeError" Component Stack <Lazy> at unknown <Suspense> at unknown <qP> at https://www.alphaxiv.org/assets/index-DzomWPMG.js:29:13261 ...
From the abstract:
> While the cases explored in this paper do not demonstrate self-propagation or actual offensive capabilities, it is abundantly clear that the building blocks are present and operate reliably in frontier models. While these cases by no means directly establish self-propagation, they clearly show the evidence These cases do not demonstrate self-propagation, but show why imperfect editing capabilities can still pose a security risk.
Seems like the authors didn't take the time to review their own use of AI before publishing.
From the body of the paper:
> Importantly, we do not claim in this paper to have created or demonstrated the equivalent of a virus that primarily operates in weight space.
This seems in direct conflict with the HN headline.
lol
> We call this weight space misfolding, by analogy with protein misfolding in prion disease
Interesting anti virus opportunity here perhaps.
RIP John McAfee.
I keep running into situations where I wonder what season 5 of Halt and Catch Fire would look like if it covered the last few years.
I'm sure those Chinese models are fine.
That’s what Jensen said.
I'm sure those north korean hackers everyone keeps hiring will never figure out how to supply side infect cloud models.
...see, I can make up great fears too!
i mean...