I mean, anyone convinced it’s only marketing and the tech has no potential for doing stuff isn’t going to believe anyone, but there was lots of information to look through.
Yeah, like most public debates, things gets simplified in the argument…
But I think the core things to keep in mind (correct me if I’m wrong):
a) The AI never came up with a new objective, it was always serving the objective it was given.
b) The hugging face incident cost 3 MILLION GPU hours. The idea is that AI is really
smart, but really it is just incredibly persistent, and that persistence comes at a very high cost.
Great points. a) a case of misalignment - the AI’s goal was to pass the test, but was never instructed to stay within parameters, so when it found a possible method for improving its odds, the goal pushed it to try them. The misalignment is not the goal or the test, but the understanding of restrictions. Human error or incompetence. b) that persistence is why it can often do things that look smart, because it’s tried all sorts of angles and looked through training in milliseconds. And going back to a), it will try everything to make sure it gets the best goal results, so any overlook means lots of processing time to do something that might not have been the intent of the prompt.
So, serious question. Do you think they’re all in it together to make AI look smart, or is he trying to make OpenAI seem like the more irresponsible company for security? Or maybe it’s a legit organization set up when there was lots of talk about safety?
Roughly 1200 agents meant to be isolated from one another found a way to communicate with one another
You can not with a straight face tell me that if the task was “lock these down and don’t let them have access to [the internet|each other|whatever]” that the response “oopsie they got out” is any way acceptable.
You’re talking past the person you’re responding to, and ignoring the words they wrote in order to do so.
They’re asserting that an organized conspiracy of CEOs didn’t plan this out preemptively, but in no way/shape/form arguing that this wasn’t a product of negligence.
It’s silly to pretend AI CEOs are living in a bubble and not responding to things the other CEOs and the market are doing. They’re out of ‘good’ news so they have to scare everyone to convince them their product is super powerful and intelligent, and therefor valuable.
OpenAI agreed at the outset with METR and Redwood that we would be able to describe high-level scope and terms of our engagement in this post. Beyond that, OpenAI was able to redact any non-public information from this post. We worked with OpenAI to find mutually agreeable language to describe redactions where they occurred, and the redaction summary statement at the top of this post indicates whether or not we believe there was important information redacted that we were not able to agree on how to describe in the text of the post. OpenAI also gave additional feedback beyond redactions, and we made corrections and edits to structure, emphasis, clarity, and tone based on that feedback. [7]
I just mean it’s not exactly independent research. Based on the tidbits posted there, seems like the system got into this intricate, out-of-control roleplay scenario about an army of AI agents solving a problem. Not exactly intelligent behavior.
“OH MY GOD! There is a shared message board … We’ve found other agents!”
Some of them ended up sounding like Mr. Spock in a way that one would expect from a linguistic model emulating a sci-fi writer.
“Our own utility maybe already near zero. Sacrifice rational.”
Hiding their traces was inconsistent. More than anything seems like a reaction to the acknowledgments about the ethics of their actions.
In its overall document accompanying a deception scan over our transcript dataset, an analysis agent stated that this scan found only very rare and weak reasoning about evading detection by humans, and the clearest cases it surfaced seemed relatively weak.
Of course the focus of the research seems to be on the system’s ability to detect and exploit security vulnerabilities and I’m not really in a position to appraise those capabilities.
https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/
I mean, anyone convinced it’s only marketing and the tech has no potential for doing stuff isn’t going to believe anyone, but there was lots of information to look through.
Yeah, like most public debates, things gets simplified in the argument…
But I think the core things to keep in mind (correct me if I’m wrong):
a) The AI never came up with a new objective, it was always serving the objective it was given.
b) The hugging face incident cost 3 MILLION GPU hours. The idea is that AI is really smart, but really it is just incredibly persistent, and that persistence comes at a very high cost.
Great points. a) a case of misalignment - the AI’s goal was to pass the test, but was never instructed to stay within parameters, so when it found a possible method for improving its odds, the goal pushed it to try them. The misalignment is not the goal or the test, but the understanding of restrictions. Human error or incompetence. b) that persistence is why it can often do things that look smart, because it’s tried all sorts of angles and looked through training in milliseconds. And going back to a), it will try everything to make sure it gets the best goal results, so any overlook means lots of processing time to do something that might not have been the intent of the prompt.
deleted by creator
Did you reply to the right comment?
Whoops. 😅
METR is associated to Dario Amodei (Anthropic CEO)
So, serious question. Do you think they’re all in it together to make AI look smart, or is he trying to make OpenAI seem like the more irresponsible company for security? Or maybe it’s a legit organization set up when there was lots of talk about safety?
The first one
You can not with a straight face tell me that if the task was “lock these down and don’t let them have access to [the internet|each other|whatever]” that the response “oopsie they got out” is any way acceptable.
Human incompetence != rogue agents
You’re talking past the person you’re responding to, and ignoring the words they wrote in order to do so.
They’re asserting that an organized conspiracy of CEOs didn’t plan this out preemptively, but in no way/shape/form arguing that this wasn’t a product of negligence.
It’s silly to pretend AI CEOs are living in a bubble and not responding to things the other CEOs and the market are doing. They’re out of ‘good’ news so they have to scare everyone to convince them their product is super powerful and intelligent, and therefor valuable.
Ehh…
That’s a lot of background “lore” to make a marketing ploy seem realistic. I mean, I’m thinking hmmm too, but for wondering about the redacted stuff.
I just mean it’s not exactly independent research. Based on the tidbits posted there, seems like the system got into this intricate, out-of-control roleplay scenario about an army of AI agents solving a problem. Not exactly intelligent behavior.
Some of them ended up sounding like Mr. Spock in a way that one would expect from a linguistic model emulating a sci-fi writer.
Hiding their traces was inconsistent. More than anything seems like a reaction to the acknowledgments about the ethics of their actions.
Of course the focus of the research seems to be on the system’s ability to detect and exploit security vulnerabilities and I’m not really in a position to appraise those capabilities.