First phase of embedded assessments at OpenAI
openai.comWe're in a surprising narrative where Anthropic is pushing forward and oAI is slowing down for safety reasons.
There's a pretty interesting matrix of possibilities here if you consider who has better mechinterp, if that's actually helped alignment, and what the internal drumbeat is around cash and IPO.
If you think your safety problems are existential, then you MUST halt and do all this. And, if you think someone else's safety problems might kill them, suddenly, without warning, then you have time to slow down and do this right.
I think if I had to pick a company to be in charge of, I'd pick Anthropic right now, given only an outsider perspective. But, I can see a lot of reasons why being a fast follower around IPO time might give comfort. Anyway, I'm interested to see some debriefs in the next few months as to what's actually going on right now, and see what the next oAI model quality is like. We may well see some pulling away from one or the other group in the interim. And it could well be oAI.
This is very surprising, and the Anthropic IPO is muddying the waters a lot. If Ant truly believed frontier safety was existential, they should slow deployments when their counterparty has shown the willingness to do the same.
As I've written about, the game theory there is really tough. You should convince everyone else to slow, so that you can not slow. And in fact even if you think everyone should slow, it's a zero trust game, so anyone not slowing means you can't slow. Ergo, calls to slow are always disingenuous.
There are plenty of real people making plenty of non-disingenuous calls to slow, so something in your analysis is off!
I think it’s always been this way. Does openai even allow remote control from mobile to Linux cli yet ? Where Boris has been recommending auto mode for 6 months. If you watch Dario’s safety talks, he essentially defines it as normal development… (ai doing what you tell it to do)… Claude is the autohotkey of ai. And I am a big sama fan…
Liability reasons. So the difference is likely how many lawyers they have and Anthropic believing "good intentions" are all they need.
Can't we just use the nane 'on-site' : it's a lot clear and comes without the baggage of the military-derived 'embedded' term.
> it's a lot clear and comes without the baggage of the military-derived 'embedded' term
"embedded" is just a term, it is not "military-derived"
Some webpages have embedded social media feeds on them. Your dog might have a tick embedded in it. You might have a PC with an embedded math co-processor.
You are missing the definition [1] from the page you linked. Typically when people talk about "embedding" a human being, they are operating under this definition.
A Forward Deployed Engineer (FDE) is has also been called an Embedded Engineer for a long time now.
Although.. I'd argue a FDE is far more militaristic in linguistic background than "Embedded" anything...
TACTICAL FORWARD DEPLOYED COVERT SOLUTIONS ENGINEER (read: we send you a nerd to sit in a cubicle and bill him at $500 an hour)
> Can't we just use the nane 'on-site'
Or the well used term 'consultants'; though if I had linkedin I would definitely add availability for embedded assessments
> Training did not incentivize subversion.
Training always implicitly incentivizes subversion. That's the whole problem. If you could establish this, you would have solved alignment.
> Reasoning behind subversion would be detectable.
There is no reason for subversion to be apparent in the reasoning traces, and in fact exactly the opposite is the problem - since only subversion not caught in the reasoning traces is rewarded, the models are implicitly trained for undetectable non-reasoning-trace subversion.
I’m sure people will pick holes in this, but it’s hard for me to see this negatively
This is so backwards and guardedly formulated that at first I thought it was something like the Google box (Google Search Appliance (GSA)) - for AI - that openAI were deploying agents (of the AI kind) to audit companies.
That might have been interesting.
But this is basically a hobbled external audit?
Why hobbled? Seems like an external audit with more access to use of internal tooling and resources.
Yes that’s nice that you’re thinking long term about these things, but we just want to know: how far off is Astra 6.2? I’m sick of paying exorbitantly for Fable 5.2
It seems that post-hugging-face-incident, OpenAI has become a culture of safetyism, rooted in fear of public backlash.
The problem: anthropic has had this culture from the start, and they will deorbit all of the surrounding safety talent into the company, leaving OpenAI on the back foot in perpetuity.
OpenAI's original techno-optimist culture is what had them in front, and it was a core differentiator.
I'm not sure I'm excited about two firms playing alignment gods. One is convinced they are bringing in a new, correct god; the other is terrified that they may accidentally birth one in any capacity.