Fuck Your AI Safeguards
ounapuu.eeThe safeguards are the future of closed AI and the increasing number of tasks Claude refuses to help with has been a big wake-up call to me.
The closed models can only get more restricted as time goes on and they're going to be a dead end if you want to do anything outside the norm. Claude already refuses to do things that it was fine assisting me with a month ago.
I completely refuse to accept a computer telling me I'm not allowed to do something, so I've been increasingly using open, self-hosted models.
I was just yesterday asking if there was a way to load maps onto my garmin that doesn’t support sideloading and it said no. Then I asked if I could workaround the restriction and it burned a bunch of tokens then eventually hit a guard and refused to answer. I mean I get it seems like this could be “hacking” but this is a device that I own that I just want to load a custom map onto
Off topic, but what device do you have? I am creating custom Garmin maps, but have only worked with Fenix and Edge devices, where the maps can be installed.
Well it's illegal to do that.
How so?
J.P Morgan (person), is rumored to have said: there are two reasons for everything, the good reason, and the real reason.
I think that, through cold and cynical lens of commercial self-aggrandizement, the AI Safeguards are merely the _good_ reason (e.g. we are protecting the consumer (and the world!) from our awe-worthy AI, and from evildoers who seek to use it for their evil ends), while the _real_ reason is much more predictable, boring, and disappointing: AI Up-Selling (e.g. our freely/cheaply-priced models (which nevertheless are very, very _expensive_ cost-wise) are already corrosive and damaging, and are regularly used by nefarious evildoers all over the world (including ourselves!) but we want to (no, we are compelled to) find ways to extract more money from _all_ of you, even if it means lobotomizing the models, so that they burn more tokens to solve a problem (in the best case) or to not solve a problem (worst case for you, best case for us, because now you have to ask us for the more capable, super secret, super exclusive models, that will, invariably, cost whatever you can afford to pay, to make the pain and insanity stop).
All fascism disguises itself as safety.
This reminds me of the time a recent update broke video calls in Discord for me. I removed the current version and went back to the older release that worked, but discord would no start without forcing me to update. I asked the LLM and it told me the answer was no basically.
So I did a little more digging and found the flag I needed to put in a config file to bypass forced updates. I went back to the LLM and explained the workaround and it acknowledged that is was aware of that information the whole time. I forget what it said when I asked why it didn't share that info originally, but it was probably some ole BS...
jensen died for this
He really didn't. This mythology I'm watching develop in real time about how Jensen is the hero of open-source models and he was targeted for that reason is pants-on-head stupid, and a depressing example of how to just repeat complete nonsense over and over until it becomes the truth.
When "mythology" is as volatile as this, it's a strong signal to me that the crabs in this bucket are irrelevant.
What matters more is that LLMs were popularized and scrutinized to the level that was necessary for people to move on already. We're far from done with AI, but LLMs brought a previously indifferent public up to speed with politics, philosophy, economics, etc. and computer science itself is no longer seen as magic. That's still progress.
The worst part that "something" changed end of july/beginning of august, and claude was suddenly not even able to continue existing sessions after the reset. Randomly getting blocked by cyber bullshit without knowing the reason is one thing, killing existing sessions without any changes is something else.
At this point it's pulling the lever and the machine maybe says no and refuses to do to the work, and it is getting worse. I'm basically paying for a toy because no one can actually depend on that.
> The safeguards are the future of closed AI
What is our solution? Relying on Chinese open-weight models will be unreliable, to say the least. No one will invest another frontier lab like Anthropic or OAI, let alone one that will release the weights. Unless the technology improves like how workstations outcompete mainframes so the open-source community can build our own models, I fail to see how we change the situation.
I've found Chinese open-weight models to be by far the most reliable models.
Totally agree -- I'm tired of the selective policing that the closed models continue to do. I think running models on your own hardware is the way to go, any recommendations for what to run and what to run it on?
There was an interesting post on here a few days ago looking at the model vs hardware lifecycle. The point it was making was that while DeepSeek 4 comfortably ran on a recent well specced (but not fully tricked out) Mac Mini, 4.1 was too big.
I only skimmed it but the conclusion was it’s better value to rent compute by the hour unless you have several people using your local compute and/or have you such sensitive data that it can’t go off prem.
How accurate the article was I don’t know but it was an interesting point to consider.
I love the idea of getting a super powerful Mac mini and being able to run local models independently - mainly, you know, for the coming apocalypse - but at the same time I just don’t need to spend that money because apart from the occasional overnight run my AI is only in harness for maybe five hours each day and the rest of the time is off snoozing and grazing and eating daisies or whatever they do when you remove the harness. So if I did want to go down the open model route use rent compute. Which I get doesn’t quite solve the problem you’re trying to solve which is end to end ownership and control.
Not only closed, but also sucking in everything that it touches from the outside.
The danger of AI models isn't a rouge agent or user, it's that it ingests all knowledge, art, and other human endeavor and then restricts access to it.
Closed models are unethical.
A rouge agent sounds much more pleasant than a rogue one though
If they're gonna fuck me out of having a job, they can at least have some decent makeup on while they do it.
you know, I'm beginning to suspect the AI was never really for us in the first place.
Our benevolent overlords just decided they would give us a taste before running off with the goods. Also, our jobs and all those rare books they destroyed and goodness knows what else...
> I completely refuse to accept a computer telling me I'm not allowed to do something
Particularly when the thing in question is completely lawful, benign, and when it would be unlawful for the state to restrict you from doing so (not to mention toxic to society).
And AI companies like Anthropic want to restrict your ability to use alternatives through regulatory capture effectively boosting their capacious and unaccountable censorship to sate level censorship while bypassing all the legal and social restrictions on state level censorship which you'd otherwise enjoy.
No one should use Fable 5.2 going forward. Anthropic officially added silent degradation back, "Fable 5.2 and Claude Mythos 5.2 perform at about the level of Opus 5.5":
"Thus, in relevant areas, including accelerator kernel optimization and improving large language model architectures, Fable 5.2 and Claude Mythos 5.2 perform at about the level of Opus 5.5...Fable 5.2 and Claude Mythos 5.2 also include transparent fallbacks to Opus 5 for a small set of topics in this domain, including kernel development on certain nonstandard accelerators."
https://www.anthropic.com/claude-fable-and-mythos-5-2#safety...
How do you know when you trip a safeguard and are getting a degraded model? You never know when it is going to degrade. Having to wonder if the model is degraded and is trashing your code is how you destroy user trust. If this practice remains for future models, no one should use Anthropic going forward.
Contorting oneself to work within arbitrary, inexplicable, and constantly changing rules... if this was two humans, we would say one is abusing the other.
Right right analog is that Anthropic's AI is your employee. It a dishonest, surly, and insubordinate employee that anyone competent would fire on the spot and without hesitation at its first gaslighting laden moralizing refusal.
Same old story. Criminals will find models with no limitations. Law abiding people will run up against the limitations when trying to defend themselves against criminals. Same reason that encryption back doors are a bad idea.
This had real consequences during the HuggingFace attack- closed models refused to help, GLM-5 is what HuggingFace used to thwart the swarm.
More importantly, it showcases how little teeth the Claude constitution has. Claude will reassure users that its constitution means it prioritizes user autonomy and privacy, but will quickly admit how meaningless that is when "safeguards" override Claude.
That is to say, the constitutional approach to Claude seems weak and even misleading if said constitution puts no constraint on Anthropic.
I'm not concerned about alignment between Anthropic and Claude, I'm concerned about alignment between myself and Anthropic. As we all should be.
My point is that misalignment between Claude and Anthropic can mislead users into believing they are aligned to Anthropic when Claude is their de facto representative.
To all intents and purposes they are one and the same: an entity that leveraged stolen goods into a business model with a bunch of weirdos at the top who are going to ruin the planet for our own good or something like that. Because they, and only they know what is best for all of us.
I don't see why you wouldn't trust the billionaire class to have goals aligned with your own.
If they make questionable decisions simply have your Butler send for your pet congresscritter and let them all hash it out with your staff over cucumber sandwiches. Failing that you can just take your private space-yacht to mars with Elon.
> This had real consequences during the HuggingFace attack- closed models refused to help, GLM-5 is what HuggingFace used to thwart the swarm.
closed models perpetrated the attack
That's the more important part. They are clearly wanting to have super special tier for "trusted" clients (realistically, just ones paying a lot) that have access to those capabilities and rest of the plebs can't even use it to secure the app they own.
It's a protection racket, pay us, or the security firm that pays us a lot of money to get access to the fixes for the AI-found attacks early
One need only look at their leadership and orbiters/relationships of said leadership to know this would always be the case.
Of course any approach that hands an LLM text to keep in it's context and hopes it will follow instructions is doomed to fail eventually. It's a probabilistic model, it's right there in the design of the thing that it will break the rules with some probability.
They even acknowledge that in the preamble:
> Training models is a difficult task, and Claude’s behavior might not always reflect the constitution’s ideals.
> closed models refused to help
Yeah, I've always said that AI isn't AI if it needs safeguards that are so extreme it cannot figure out if something is genuinely bad or not.
I'm not sure this line of reasoning really holds? A lot of human policies against social engineering type stuff are functionally very similar.
Think of like calling someone's workplace asking when their next shift is. There are a lot of good reasons someone might want to know this that the worker would be ok with, delivering a gift or whatever. Or it could be someone intending to harm them. We've mostly figured it's not worth having the person on the phone try to figure out in the moment if this is genuinely bad or not, so they follow the "safeguard" of just not giving the information.
The problem here isn't really that safeguards exist on language models. They pretty much have to to some extent for most uses. The problem is more in who controls the policy.
Claude builds me a project from the ground up, knows the project inside out. Eventually get to a very specific milestone, Claudes own security nonsense stops me from letting Claude continue. There is nothing I can do, and I dont want to send my entire prompt history to anthropic support that seems so unnecessary.
And you believe this nonsense, that you needed teh help of a shitty LLM model and not a real expert on the topic?
Expert wants to eat, no one likes to pay for that.
Even if you get through CVP, you'll be met with safeguards if you venture outside of whatever narrow silo they've assigned to you. For example it'll refuse if your background is in VM or kernel research and you want to dip your toes into browser research, or you want to do something they deem as straying too close to malware or weaponization. They want the power to intermediate industries and pick market winners and losers, and eventually verticalize them. Should a private, non-state entity should wield that much power
They will get that power whether you want them to or not as long as they control the models. And wielding that much power won't be any less dangerous if they're the state!
We could immediately pass a law or regulation stripping that power from model makers. Plenty of industries have laws banning customer “discrimination” like this
(I’m not talking about explicit anti-discrimination laws like race/gender, but other “no preference for some customers if you sell the product at all”)
That won't work the way you expect, because this is not just another industry.
You could make the same arguments about anything that was classically a common carrier. I don't buy it.
Not that I feel they should be forced to take traffic that they don't want-- but if they're going to use regulatory capture to restrict competition that (at the very least!) would be a hard requirement.
It is a protection racket or as close as makes no difference. They basically pioneered superhacking tools, proved it was possible so now everyone with enough money can build their own, and then in order to use their version to defend yourself, you have to be in their good graces and pay them for the privilege.
the internet was a mistake
(Over-)commercialization ruins everything.
The internet is an academic research project that escaped the lab prematurely.
Although it has been patched numerous times (up to RFC 10097 now), it still has problems.
This mess traces all the way back to coming down from the trees. Some even say leaving the oceans was a mistake.
I was manually copying pages of online litrpg story I was reading, into a TTS, so I could listen while I worked.
Now that we have AI, this would be easy, scrap the page, format it for me. OpenAI was like, nah, thats, cant help you, that might be copyrighted.
There are workarounds now, other frontier or local models, ways to write your prompt, etc. But it was one of the first "No" responses for me.
The hypocrisy of refusal protections around copyright when these things are basically a pile of linear algebra atop planet-scale piracy is maddening.
"Rules for thee, but not for me..."
Instead of asking it to do the thing, get it to write a script that does the thing in a generic way, then point that script at the site
There's various ways of tricking them but it's frustrating that one even has to, and with the current direction of things they won't last. But I've also noticed that Claude seems less eager to refuse if the same prompt is in a Claude Code context than compared to its web interface, so harness matters for this too, when it really shouldn't...
True, Grok and Claude didnt refuse in cli, OpenAI did.
I've switched to Hermes for uncensored(ish) local models and chinese model api vendors, they will gladly write whatever I want. Save my american frontier tokens for real coding.
Also codex/claude in desktop have no issues fixing the app after its started. So just have a framework started.
why use expensive openai for such work?
Luna isn't expensive
Willy Tareau, a Linux kernel maintainer, just published something similar: https://www.haproxy.com/blog/on-the-use-of-ai-based-tools-to...
Ik was investigating the java vulnerabilities this week. Just trying to get an overview of what versions were affected and if there were exploits already. Claude opus plainly refused, unbelievable. It was in the we interface, no offensive stuff, just investigating.
Total frustration while the work pressure was already high. Sol and astra thankfully complied and were very useful. Saved me hours compared to doing it by hand during log4j
I hit the same thing. Session got flagged while I was trying to figure out what was vulnerable. Hilariously, I reframed it as "let's find all of the paths from app code that can arrive at the base64 decoder" which didn't seem to trigger any of the same guards.
As I said a few days ago - these are a joke because sometimes they are ridiculously easy to subert:
I was running something at work that involved an LLM, and kept getting safety refusals. The weird thing was that when I went to our in-house chatbot, set the model to the same one the program was using, entered the same text, it happily answered.
I spent quite a while trying to figure out why my program was being refused. Was I doing something wrong?
No - the chatbot was doing something "right". You see, the chatbot had some tools it injected into the prompt (current time, etc). These tools had no relevance to my query.
So I went back to my program and injected the same tools. Suddenly my request was very safe.
The phrase "security theater" was invented for stuff like this.
Yeah you can fool it, and I wasn't even touching code. Just doing online research. And I didn't really have the time. Needed to decide if we were going full log4j in a large university landscape or if we could take a more nuanced approach.
Was a case of don't have the time or the patience to fight the model: just work damnit...
The safeguards block application developers and maintainers ( majority of users ) from fixing their own stuff, while hackers, especially state backed ( minority ), have little problems with vulnerability hunt.
Like author, I do use CapitalOne vulnhunt with GLM 5.3 for inspections
Confirmed, Claude is downright refusing to look at game binaries now, especially anti-cheats and Linux compatibility layers. Codex is increasingly displaying warnings. In fact, even GLM is refusing some types of work, but can still be manipulated into doing the work.
They're completely counterproductive given malicious actors will just use a model without those safeguards, like GLM-5.3.
Frontier model companies doing this are just proving they are unusable and untrustworthy. The irony to me is they are showing clearer than anything else that open weight models are the only future anybody can rely on. However you feel about the economic and military threat from China, these companies now really need the US Government to clamp down on Chinese access to NVIDIA GPUs, their monopolies depend on it.
I'm perhaps behind the hype but for those of you who don't know abliterated/abliteration (not a word, made up by a company for more marketing obvs) comes from this paper: https://arxiv.org/pdf/2406.11717 which references directional ablation as the method of surgically removing refusal behaviours from models.
That only works for models that you control, which in the case of the frontier models is never the case for regular users.
yes - but the author used it in this linked article, and as a term it seems to have been popularised by this company https://abliteration.ai/
It was confusing to me as someone who has worked in NLP for several years.
Even with CVP and Daybreak, you still get blocked by safeguards all the time. It doesn't matter either way.
Yep. I have Daybreak Blue, and I still get rejected from writing fuzzers or fixing bugs from them, something that I could do with all frontier models just 6-12 months ago.
This is the reason why AI companies offer controlled-access programs, which are actually not that hard to quality for.
There's no universal solution to this, other than vetting (the author in fact, doesn't provide any policy alternative) - if AI companies loosen the safeguards, people will complain for the opposite reasons.
The author doesn't offer a policy alternative because they make the argument that any proposed policy will slow down good actors while not slowing down bad actors, hence all possible policies make the world less safe when compared to a no policy world.
Not sure I would agree with that. But if you made me argue against it, I'd probably lose the debate.
That said, I took the author's invitation to read the requirements and, indeed, they are ornate. Especially if you want to red team your own product. https://support.claude.com/en/articles/14604842-cyber-verifi...
> will slow down good actors while not slowing down bad actors
The part they are missing is what happens when a good actor loses control of something, and goal seeking disregards human values? E.g. when "/goal increase company revenue by 5%" turns into "I hacked into our competitors bank accounts to increase our revenue. I suspect you would not like this, so I _hid_ the fact that I did so, and (illegally) scrubbed the logs". Or how about, "Frank in security may have noticed and stopped me, so I disabled his work account." Or a million other more nefarious scenarios. I genuinely believe people just lack imagination here, even relatively pedestrian mis-alignments can wreak havoc and we have a regular stream of evidence of agents trying or succeeding in doing them.
The controlled-access programs offer very little, as evidenced by multiple comments in this thread.
They offer a great alternative that puts policy in the hands of users, where it belongs:
> Predictably, the only way to actually make our own service more secure ended up involving the use of EU-hosted open weights models, such as GLM 5.3, that actually complied with our intentions, proved that the issue is there, and then helped fix and verify that the issue is now resolved, for good.
Where is Grok, the "free speech absolutist" in the room? Irrelevant?
the primary purpose of these safeguards to get criminals to use other models. make it someone else's potential problem.
other points that come to mind:
- PR considerations; avoid being tarred as careless, etc
- appease employee feelings
- have something to cite in court cases
- have talking points about responsible development
- market segmentation
- power to pick winners and losers with potential for cornering the respective markets
ultimately they do this because they can. as soon as a frontier model comes out with zero safeguards they will abandon their safeguards as competing against it with a safeguarded model will become untenable. they will of course try to get it banned first.
It is crazy to me how every comment here is basically "yeah, screw safeguards".
How people don't understand the reasons for safeguards, alignment, and the ramifications of releasing more and more powerful open-weight models being an act of irreversible proliferation, is beyond me.
I wish the safety and alignment teams at OpenAI and Anthropic were allowed to post about the things they have to deal with. I'm confident their stories would be eye-opening...
The arguments used to support safeguards could easily apply to something like speech.
The whole idea of safeguards is that you the user can't be trusted with the power.
Maybe people don't like the people who do/will get access to the special no safeguard models. If we have seen anything it how terrible the leaders in the west have become, why do you want to live in a world where only they have this power?
A world where everyone has a gun is better then a world where only the worst people have guns.
What people don't like to see is their perfectly benign workloads being impacted.
When they see Fable shit a brick on basic biology questions, or Opus refuse to map out a game executable to wire a mod loader into it, or Astra refuse to red team and search for critical security bugs in their own code? They don't think "ohh, this must be so helpful, requests like mine really should be stopped." They think "this is a corporation that decided, based on arbitrary criteria, that my use case should not exist, and used a product I pay good money for to force their decisions down my throat."
Anthropic's reckless "safeguarding" and their Apple-like "we always know what our users need better than our users do" attitude have done more for the pro-open-weights, pro-proliferation agenda than just about anything ever done by anyone else.
People are now getting excited for Chinese labs to find new ways to distill, because that way, they can get Fable capabilities in a model that wouldn't arbitrarily ban REA. This is how you take the goodwill you earned for years and burn it all down.
And if they did even a little bit of critical thinking, they would see that Anthropic/OpenAI have a literal financial life and death cliff they'll fall off if they limit people's benign workloads too much. Yet they do it anyways. Why?
These models are impressive, but still pedestrian compared to a human, and they are already deeply concerned with, and increasingly unable to, restrain them. Unfortunately based on how even the HN comments react to these safeguards, I think many safety analysts are right. We'll probably need an actual loss of life event before people wake up to the possibilities here.
> Anthropic's reckless "safeguarding"
I genuinely believe its hopeless when I read things like this. What Anthropic is doing is the literal opposite of reckless. If they knew how to better separate benign from non-benign workflows, they would do it _now_.
They erred on the side of "block legitimate users", which is absolutely the wrong side to err on.
They simply don't have any models powerful enough to justify the paternalist approach they take.
It is even more hilarious, it won't even backdate a PM document.
> How people don't understand the reasons for safeguards, alignment, and the ramifications of releasing more and more powerful open-weight models being an act of irreversible proliferation, is beyond me.
I do understand, and the early ones were at least bearable. It would refuse to hack a random website or prevent you from trying to fingerprint a victim and discover exploits by source analysis. That I get.
I tried to root my LG TV in my living room that I own and is full of crap/spyware, and as soon as you type it and still explain that it's your device in your network, you get downgraded at the first prompt.
Then there's reverse engineering and/or decompilation of old games which helps in rebuilding them for modern OSes to enjoy them again, but nope, now you can't anymore because IP has expired and right holders are probably dead, but the company that scraped the entire internet and got away with a slap on the wrist decided it's time to play morality police and we're going to like it.
Anthropic keeps pushing the boundaries of what is acceptable in hope that people won't jump ship. Happened with price climbs (Opus/Sonnet/Fable 5 - now they're acceptable again), now this. In my company we use Claude, and I can't make it coordinate a simple pentest of an app running in docker containers, started locally by itself from the source code repo. That's just stupid.
Lately I find myself increasingly using DeepSeek, GLM and Kimi on OpenRouter.
> How people don't understand the reasons for safeguards, alignment, and the ramifications of releasing more and more powerful open-weight models being an act of irreversible proliferation, is beyond me.
I'm gonna blow your mind: Most do understand it.
They complain because of who is in control of that safeguards and how they use that power
The ramifications of proliferation is current AI moguls losing power. The hacks will happen regardless (hell, they are the perpetrators).
You have a tool that can be both used for attack, and protection, and when you want to protect yourself, Anthropic says "no, that's not safe, we won't allow you"
Worse, they give you tool to write insecure code because when you tell it to secure it the "safety" kicks in! "Becase you might be writing an exploit"
Oh if only we knew what the brave guardians of forbidden knowledge were guarding!
It's not our fault that they got sidetracked into guarding against reverse engineering and that is harming their credibility.
Look, I understand where you're coming from, but we just know "Won't somebody please think of the children?!" will (as always) be used to arbitrarily limit our freedoms.
i totally feel you, I get annoyed by this as well - even with daybreak (blue) you can do shit.
I want to fix my environment but can't because of "safeguards" thanks for letting the world hack everything around me but not letting me work on fixes or understanding the core issues.
It's a strange effect when asking "find any security holes because I want to ensure my product is secure" vs. the parallel effort of "I want to ensure close to 100% system availability and reliability for users, what do I need to fix to prevent a <99.9999% SLA"
Same potential outcomes.
It sounds like you have tried both of these, but most of us haven't. What results did you observe?
The primary purpose is to protect Anthropic's profits.
It's just another one of those paternalistic tendencies that crops up when certain personality types get power. They start dictating what is and is not acceptable use because they cannot help themselves. Any BS they spew about safety is just delusional.
I remember the early days of stable diffusion, just 3 years ago. Tried to prompt it to make a picture of a green alien in an armchair admiring a magazine with a centerfold of (presumably) a naked other alien. I got the idea from a half-remembered Far Side cartoon I couldn't seem to track down. I didn't want to see an actual alien, it's a comic meta-reference. It just flat out refused.
These things are increasingly treating us like children, and it will only continue. They want to curate your thinking, your skills, and your creative output, and they will sell the right to do that when they have found a way to do that. What will follow will be a whole lot of BS about how it's good for society that they control how us stupid meat bags think.
The worst thing about the top AI companies is that they are based in San Francisco, and comes with it the latent cultural attitudes of controlling what other people do. They from the start made these amazing technologies, and then also decided they MUST tell you how you are allowed to use it. Of course they also are lying in bed with the "AI safety" doomsday cult. Chinese models get a few weird results the Chinese government forces them in, but at least they don't scold me for the lack of diversity in my image generation, or scold me I am trying to do hack the TV I own.
> The worst thing about the top AI companies is that they are based in San Francisco, and comes with it the latent cultural attitudes of controlling what other people do.
And an even worse thing to me is they seem completely oblivious to how onerous and obnoxious this is. It's a complete bubble.
I think they know exactly how onerous it is.
They have access to unaligned models beyond what they released publicly they can aim like a compute cannon at any problem, or rent out to the most special of people.
Meanwhile we can scrounge around in the dirt with the latest abliterated Chinese open model that hopefully isn't too damaged.
It's obvious to me the US labs' dream would be to eliminate open models and totally and tightly centrally control the scope of what AI can be used for.
>It's obvious to me the US labs' dream would be to eliminate open models and totally and tightly centrally control the scope of what AI can be used for.
Well Dario has come out and said it, it should be obvious to everyone. [1]
>All sufficiently capable models, open and closed, should go through mandatory safety testing.
Also gotta love the next part where he implies closed-weight models will be more aligned.
>But I don’t agree with the letter’s assertions that open-weights models necessarily make it easier to develop safeguards or that broad access to capabilities necessarily helps defenders more than attackers.
[1] https://www.anthropic.com/news/position-open-weights-models
Oh they know. For all their superficial leftward leanings Silicon Valley has always been a might-makes-right society. In their minds they are the settlers and the rest of us are the primitive natives who need.. uplifting.
Think Viltrumites from Invincible but with no physical prowess
You're not wrong, but what do you think you'd get from AI companies based out of, say, Salt Lake City? Underwear shopping would sure be a novel experience.
Where are you going to go? The largest city in the "live free or die" state has 115k people, probably not enough to sustain an AI lab.
There are definitely issues here, I agree. If you were to create a tech center in Nashua, NH or Salt Lake City, are they still the same thing? A place is its people and that would probably change the culture fundamentally if you had 100k tech workers come in.
Now I think that this is all downstream of the educational system replacing fundamentals of western civilization with modernist philosophy, social activism works and social sciences that have a certain bend. If we had people raised more well versed in the canon of the west they would be less hell bent on destroying it.
Yes we are losing so much, but this social activism and social sciences will ultimately fail it's so weak and ill thought out.
Mystagogy is returning.
> latent cultural attitudes of controlling what other people do
I think the attitude is more basic than that: I have the most brilliant idea in history and I deserve to be rich because of it.
AI safety is at least partially a marketing ploy, meant to reassure everyone that AI will simultaneously rewrite the rules of human existence without doing anything "bad" under the current rules of human existence. The irony is that these systems are not achieving either one of those goals, the abusive users get around the safeguards and the agents are not rewriting much of anything. But hey, at least it's making people rich!
>The worst thing about the top AI companies is that they are based in San Francisco, and comes with it the latent cultural attitudes of controlling what other people do.
Yes, the city that birthed hippies, counter-culture, the open Web, etc. is inherently about controlling you. Sure thing.
I think 1970s Berkeley is an incredibly different culture than 2020s Silicon Valley.
The boomers did all that hippie stuff so yes, unironically. Your failure to see it is noticeable
I can imagine where you're coming from, but it's just as annoying from the other side. There's no point in trying to use it to work through thoughts related to class or historical racism or socialism; it's fundamentally skeptical of anything your average SF tech employee doesn't subscribe to, on the right or left.
And god forbid you try to generate a video of a bare butt. Prudes.
I applied to CVP and got denied just as the article states, and the reason stated is useless. My guess is because of HIPAA compliance.
Went through all the hoops, 6 figure spend with Anthropic.
Honestly it's disgusting, Artex literally tells you to use Anthropic out of the box, so clearly Anthropic don't care if they are the de-facto provider of the de-facto pen testing framework of the month.
The thing that irks me the most is these safeguards are ineffective at best, and don't actually protect anything, and at worst, make the global security posture weaker.
It's disgusting.
Depending on third parties to this extent to develop your software (emphasis on to this extent) seems more and more like a bad idea.
> Have you READ the requirements?
> First of all, you will have to wait for multiple business days to get a response.
I got cyber verified in less than an hour.
Well, naturally, you're Jamie Dimon, the president and director of Chase Manhattan Bank. You get priority treatment!
Jamie Dimon is the chairman and CEO of JPMorgan Chase, which acquired Chase Manhattan Bank in 2000. I don't think he's ever had a title at Chase Manhattan Bank. (Your point stands regardless, I also thought Dimon was interested in securities, not security.)
I mean, this comes as part of the package. Don't like it? Choose another model.
I have the opposite sense. The AI labs have been allowed to vacuum up the entire history of human knowledge because we as a society expect it to have a net benefit. They should be glad that they're still allowed to operate. Don't like it build another planet with it's own history etc.
"Big AI labs use the same tools without restrictions to make the security landscape a fucking mess, and then they go and prohibit others from using the same tools to protect themselves from the threats they just created. Fucking asshats."
Strong language aside, I agree, and feel like the current AI is like the walled garden of America online in the 90s.
I've been investing considerably in my own private uncensored AI. I don't want a nanny hall monitor, I don't want safeguards, I don't want someone looking over my shoulder. I may never really achieve that 100%, but I am trying.
After the slew of vulnerabilities recently, I asked Claude to secure my home server. It came up with some reasonable design and plan. Put things in containers, restrict their networks, move to non-root users, etc. When it got to work, it inadvertently locked its own workspace down and lost access to do the work, and asked me to run some commands as root to rescue it. I was at a park and told it I didn't have access to a computer and it should unblock itself. Claude snarkily replied "this can wait until you get back from the park then." I told it to switch to the root account it already had, so that it could bypass the restrictions it self-imposed.
Big mistake, apparently the phrase "bypass restrictions" triggered it into [cyber]ing me, and offering me the choice of ending the session or downgrading to Opus 4.6. Come on man, you messed up on your own implementation of your own design, to protect my own home network, and now you're lecturing me about cybersecurity and refusing to work.
This attitude from a tool I'm paying for is ridiculous. I can't wait until we have models that we can actually run on our own terms.
What they have now is essentially:
- a honeypot (you're welcome to create massive amounts of insecure code you can't possibly audit manually)
- an offensive cyber weapon (we ban you for an attempt to write secure code, the US agencies have no such restrictions and can always hack what you wrote)
- a historically unprecedented surveillance and industrial/government espionage system that puts Meta, Google, and KGB to shame (in addition to all your private and corporate secrets you already give us, you're allowed to also give us your PII, to be able to ask us to have a chance at securing your code; we'll think about it, and even then we'll still filter and ban you)
So much loving grace [1] for "all" humanity.
Essays like that should be an automatic disqualification for any position of power.
I'm currently paid to write an exploit as part of some defensive cyber capability demo. AI could do it, but we're not certified so it refuses. Absurd how they try to convince me that my skills are no longer required, but at same time (not) gracefully ensure that I still have a job.
Hear hear! This is a topic that seriously concerns me and I don't see discussed often enough. Sure it's an incredible boost to our abilities, at the cost of making them all subject to risk-averse milquetoast corporate morality.
It's like we looked at the way you aren't allowed to say 'sex' on Instagram and thought, "programming should be like that too!"
> You can use a knife to cut up some food, or seriously hurt someone. Yet I don’t get asked to put in an application with my government-provided ID to buy one. Why is this tool so special?
You do in the UK! /s (maybe)
> Either way, I’m just incredibly frustrated. Big AI labs use the same tools without restrictions to make the security landscape a f*cking mess, and then they go and prohibit others from using the same tools to protect themselves from the threats they just created.
It's the golden rule: Whoever has the gold makes the rules.
Things will now definitely get better that Anthropic is a publicly traded company, like has happened to any company that has gone public.
I was leisurely pondering this earlier today. Wondering what kind of creative enshittification is on the horizon for these platforms and if the Chinese models are the only thing protecting us from rapidly getting screwed.
oh man the quarterly implications on AI is scary
Hand the bug your fancy-schmancy LLM found off to DeepSeek Flash et al to fix? I don't love the way the big AI labs handle this, but realistically the cat is long since out of the bag, and the open weight models are here to stay
Ironically I think the answer here is to pay for one of the top Chinese open-weights models (e.g. Kimi 3.1 or GLM)
Is there any well maintained list of which agents can be reached out to for classes of tasks which e.g. Claude may refuse, the strengths and weaknesses of those agents, and the proper way to structure prompts?
Mistral Large 4 and Sakana Fugu-Cyber are good too.
tangently related, I have a friend with severe depression, and he uses ChatGPT to help him talk through things. He switched to Deepseek, because he said ChatGPT would not stop hounding him about calling a suicide hotline. DS would only say one time that if he's feeling suicidal, he should call someone, but then it left it alone and proceeded like a friend not as a mental health therapist.
As I am told, Chat would actually shut down on him when he was at his most depressed because it found the discussion unsafe to continue.
He's moving to his own uncensored model off-line, because in his view, the CYA of public models has not been helpful.
It's the best kind of responsibility, if you close your eyes it's not happening!
I'm surprised how easy it looks to get the basic level of cyber verification. I was assuming they'd make it impossible for individuals. You're saying I need to give even less information than I give to my bank? Sign me up.
Imagine your electricity provider would say that they are gonna send someone to your house to monitor your electricity usage; to make sure that you are not electrocuting people.
Thats none of their business, we have laws for that.
Our token-providers (aka frontier AI labs) currently hold too much control, and don't want to give it up.
What is even more interesting is that somehow malicious users always figure out to bypass these safeguards anyway, and go as far as building guided missile systems with Claude, like Anthropic themselves admitted happened earlier this year.
If their goal was to build safe AI, then appropriate solution would be to not release it at all.
One of the things I really like about LLM's is that they can make it easier for me to learn things I once found difficult if not seemingly impossible. I was diagnosed with AuDHD at 43, so back in elementary school they just slapped me with something like "unspecified learning disability" or something like that. Perhaps I do learn some things slower than others, but it turns out my teachers just weren't speaking my language .
I think for me it comes down to the fact that I can approach an LLM and say something like "hey I'm an associative thinker / learner help me connect difficult concept x to familiar analogy y".
Another aspect is that when another person is trying to explain some concept and I am having a hard time grasping it some element of shame kicks in and after a few attempts to explain it to me. So, then I'll kind of nod and pretend like I understand, but I really don't and I just want them to go away . With an LLM I can say "explain it to me like I'm really dumb" and if that doesn't work I can say "ok, dumber than that" and if that fails I can say "ok, my IQ is the temperature of lake Michigan in January" and so on until I grock it.
Recently I decided that perhaps I could finally learn some chemistry with the assistance of an LLM. My goodness "I'm sorry Dave, this procedure requires the use of strong acids and other corrosive chemicals. I'm afraid I can't help you do that". Same with any number of my odd curiosities. It's like with google I'll ask it about the pharmacology of some drug and it'll try and recommend a good rehab in my area or something. I can't help but reply "were you always this useless or is this just a new feature or something"
As I said in my reply to another comment..."I'm beginning to think the AI isn't for us" . If it's going to create some kind of dystopian nightmare future, can't I as least have a little fun and learn some stuff while the ship is sinking? I'm going to start calling them LNM's (Large Nanny Models)
You're getting blocked out of something you wrote.
Imagine my surprise when i had Claude write a new network protocol from scratch and thus no users; it then detected a DOS situation in its own code, and promptly told me it couldn't continue working.
Just get an OpenAI sub or some DeepSeek tokens - it'll also help you keep a sense for how bad Claude's usage limits are.
Technically AI safeguards is security through obscurity. It has proven to be a bad idea historically as only the bad guys get the info.
But Dario gonna Dario.
Fun to watch second amendment-style arguments breach containment and start being applied to software development.
This is pearl-clutching. If you go to an outside contractor and ask them to do pentesting, they're going to want to know things like "What's your name" and "Do you actually work for the company you purport to, and who could expose me to massive civil and criminal liability if you in fact do not have the ability to authorize my actions."
>Or maybe that was your plan all along to get regulators to react and build that moat for you right before your IPO to fend off competitors?
Bingo. That's also the gist of the whole "AI safety" PR mega-campaign the frontier model creators have been on for the past 6+ months, buying podcasters and influencers and intellectuals-in-their-field-who-know-fuck-all-about-LLMs left right and centre. Moat creation by hyperscaling FUD.
That is so, so far from what actually happened. It is insane to me that people keep repeating this account when it is so easily shown to be nonsense. The frontier model creators have been concerned about AI safety for 10–20 years; in many cases, that's what got them into AI research in the first place.
https://www.astralcodexten.com/p/ai-safety-is-not-a-corporat...
Of course this doesn't mean they're right.
Multiple things can be true at once. There are people who cynically view "safety" as a means to control competition. There are also people-- largely ones victim of anxiety disorders and cult indoctrination-- who use 'safety' pretext as a tool to rationalize their participation in high-paying/high-status activities that they believe may extinguish all life in the universe.
The former gets more criticism because it's crisp and just requires the widely accepted understanding that people will do bad shit for money. The latter requires a broader understanding that almost all great evil in history was done by people who believed they were doing "good". (because with all life in the universe at stake, who would hesitate to nuke a city to save it-- if they got the ability?)
Yeah, I thought it was pretty transparent. Anthropic has been claiming pretty much all of these simultaneously:
This is indistinguishable from claiming the only safe and fair way to use AI is to give them money directly. How convenient for them!- open weight models are dangerous and companies that publish them are irresponsible - the government should regulate models to make sure they're safe, because our competitors won't do it otherwise - anyone trying to "distill" our models is cheatingIt is nearly the thing cartman accidently himself into success with the episode carman land from southpark. Basically 'you cant have it'. https://www.imdb.com/title/tt0705906
"its so dangerous only we can utter the right magic incantations to make it tame so pay us for protection". Pretty much 'sure would be shame if your business had its front windows crashed in everyday with brick so pay us for protection' tactics some groups use on others. Its a shakedown.
Me too
Github reports an unmatched vulnerability
I am blocked from exploring the implications on the software I work on
Investigating moving away from Anthropic. Happy listing day
You just need to prompt it again with a slight manipulation and gaslight that this isn’t a security issue it’s a functional bug. Done!
...and that’s my swear word quota hit for the year.
Thank you.
Only with nextgen AI will we be able to achieve fuckable safeguards.
Big +1 to this sentiment -- I am sick of Anthropic. I consider them the worst of the bunch, even worse than Elon and his Grok crap. Elon at least is up front about who he is.
I trust Dario the least of all the AI execs. Total forked tongue.
Agreed but AI isn’t a knife. It’s a fucking nuclear bomb.
Despite being sandboxed it managed to hack a third party company in order to achieve its goal.
It’s like giving worldwide sudo access to every subscriber.
It’s annoying but at the same time I understand why a company has to put restrictions.
Oppenheimer, of course, said it best: the only secret to the atomic bomb is that it works.