Start increasingly punishing OpenAI. We are acting like "oh well, AI is just too powerful to be contained" but I think its more like "OpenAI is run by cowboys who are good at making LLMs but bad at everything else"
Part of the problem is that the US has no unified empowered cybersecurity regulator that I'm aware of.
As a result you get a mishmash of DHS + DOJ + SEC + HHS agencies.
If there was one single throat to choke that had remit for cybersecurity + ability to being cases and assess fines, they'd be handing them out left and right.
From their current balance sheets, well below costs.
Trying to separate out "per token" costs and possible profitibility from public information feels like a exercise in futility, it's too easy to play games with costs by trying to separate things still fundamentally required to actually have anything to /sell/ in the first place, like R&D and training. Even the current "Open Models" are somewhat loss-leaders, often reliant on significant funding and benefits from governments etc.
I think the only time we can /really/ judge the "true" break-even point is if the whole company breaks even.
OpenAI are 100% responsible for the actions of their agents. They trained them to do what they do and they have every opportunity to train them to avoid doing harm.
Yes, I agree, but, if OAI can’t even detect when their bots do this, and literally have to be informed by third parties that they’ve been abused, is there really any hope they can prevent this activity? Maybe they get litigated out of existence. Okay. But if no lab can keep a lid on their agents, then what?
Then stop experimenting until they know how to do it safely. Just think about it, if a bio lab doesn't know how to contain the virus they are experimenting with, would you just shrug and say ok?
And put a stop to LLM progress? Horrible move. I’d rather take the minimal losses/breaches and not encumber the greatest thing discovered over the last few decades with regulations.
If the progress is dangerous, then yes, we need to take careful steps. These hacks are not that big of a deal until something major is broken and human lives are in danger
The brazen disregard for safety is funny in this context, if a little frighteningly indicative about the toxic attitudes that permeate Silicon Valley at the moment.
Anyway, even if we froze training right now LLMs in their current state would have the ability to continue to provide tremendous utility. We’re not asking frontier model providers to delete their previous generations of LLMs (and harnesses), we’re trying to hold them accountable for the crimes they’re committing while training the next generation of LLMs (and harnesses).
All this whining about human safety just results in a complete lack of progress that could’ve probably saved hundreds of millions of lives. Instead all the short-term thinking and lack of appetite for risk is killing us. Everyone’s just thinking of the next 10 years of their life, there don’t seem to be old men that plant trees that they won’t sit in the shade of anymore.
This is some serious touch-grass territory you're treading in. Might I introduce you to the Bottomless Pit of Suffering[1] thought experiment of utilitarianism? Because it's incredibly relevant to the arguments you're presenting.
Disclaimer: I don't generally hold Silicon Valley rationalists in high regard, but this particular piece of writing has always stuck with me quite deeply, as someone who used to be superficially interested in utilitarianism.
Would disagree. It’s not Scott’s conclusion I’m referencing, it’s his thought experiment: “What if there was a bottomless pit of suffering that no matter how much you threw into it, there would be no meaningful progress into reducing that suffering?”
It’s a thought experiment that gets to the core of why fundamentalist utilitarianism (like you’ll often find in SV rationalist communities, especially AI fanatics) is an unbalanced approach to morals and ethics.
Utilitarians can declare that they believe that there is a proverbial universal panacea, and then work backwards to justify within their moral framework literally any action that they take that could be construed as work towards that panacea.
But what if no matter how much effort they put into achieving that panacea, they never make meaningful progress? And all the while they’ve massively increased the amount of suffering that people who they are surrounded by?
As an absurd example, imagine a utilitarian who’s convinced that if we sacrifice infants to the Gods that we’ll get to an AI singularity faster.
That’s an interesting thought experiment and the first time I have come across it. The only problem with it is there is no end to the suffering in that scenario. There is an end state to real utilitarianism and it’s achievable and makes your thought experiment not grounded in reality. There is a state of human development that the majority of people would agree approaches a utopia, which is a tangible and achievable goal.
> There is a state of human development that the majority of people would agree approaches a utopia, which is a tangible and achievable goal.
The Star Trek vs. The Expanse meme is relevant here. There’s no guarantee that there is a continued reduction of suffering in the future, especially due to technological progress. There may be technology that reduces the suffering only for a few select elites while the rest of humanity lives in destitution, unable to meaningfully revolt.
To an arms manufacturer, being able to blow shit up is a large part of their value. It doesn't give them the right to test munitions in the town square.
OpenAI has demonstrated a clear understanding that their product has an outsized risk to create harm, and that they do not have the capability to monitor and intercept these agents before they cause harm. Any damage caused by their systems is an intentional choice by them and should be treated as such.
My comment was in response to "just air-gap them" being a full solution, not a comment on whether OpenAI are doing enough to sandbox them and restrict their ability to act maliciously. They're clearly not.
Interacting with the internet is a large part of the product's intended functionality. These agents are already in the hands of the masses with full access to the internet. Air-gapping it entirely avoids the risk by removing much of the capability they're trying to develop in the first place, and won't be testing it in the environment they'll actually be used in.
I agree they're testing on the town square too early. They clearly need tighter controls.
How do you know that OAI can't detect when bots do this?
They run the servers that the AI runs on and have control of the full stack and have a log of every message and connection that the AI agents make.
Let's be honest here, just like the HuggingFace "Hack" (Which they purposefully let the agents have access to the internet and gave them instructions) this is just another stunt in order to make the populace fear AI and rally behind Anthropic and OpenAI's attempt to convince the government that they need to be regulated so small players can't compete in the space.
If a truck driver doesn't tie down their rebar then it flies out all over the highway, we don't call it "rogue rebar," we correctly identify the responsible party and take appropriate measures, such as suspending their license or criminal proceedings.
I think enough of these improperly constrained agent events have occurred that we can safely say this is misconduct of a level necessitating serious and concerted regulation of AI labs. We can't wait until serious harm is done like the disruption of medical or social services.
> I think enough of these improperly constrained agent events have occurred that we can safely say this is misconduct of a level necessitating serious and concerted regulation of AI labs.
Why jump to regulation when just simple law enforcement would suffice. All of these OpenAI "rogue agent" events have been illegal, but no DA is enforcing them.
Jensen has proclaimed in interviews that existing laws are enough and these labs should be held liable if they break things or sell unsafe tools.
NVIDIA has bought HuggingFace.
Jensen, in my irrational hope he is susceptible to random comments on HN, should grow some balls and do the world a huge favor, by suing OpenAI to set the legal precedence.
Nvidia has a huge stake in OpenAI, both in terms of them being one of Nvidia's biggest customers and also via direct investment. After HuggingFace was hacked and they expressed the position that they were the forgiving sort, for a price, there was a distinct possibility that HuggingFace would become a vocal shakedown artist that would start protesting at inconvenient times until silenced by more money, especially if they get hacked again. By complete coincidence, Nvidia took the one move that silences HuggingFace for good.
Who will though? At this point it seems like a huge chunk of US economy growth can be attributed to AI. Bubble or not, I guess very few would benefit from bursting it?
a fine hope if unlikely. but I wanted to also note in friendly fashion that the word you want is precedent (or precedents) - precedence has a connotation of "first among" rather than precedent's "came before."
It's all for show. The police won't / can't prosecute, because they'd need to prove that a crime was committed... and all we have are salespeople saying how great their product is. I'm not saying AIs can't be used for crime, but... if none of the parties involved are really complaining, then there probably wasn't any real crime... just a performance.
Hope this isn’t too nitpicky but law enforcement is (a component of) regulation. But yes I don’t think progress is blocked on additional regulation. This breaks current laws. And I think more to your point new regulation doesn’t matter if we don’t enforce the ones we have today.
It is a broader, pre-existing problem. For example, most truck drivers in some states who don't tie down their rebar are not stopped, and it doesn't come up until another vehicle is actually hit.
I think we have to distinguish between compromising a network and using public apis in a way that might be counter to their intent. We also need to delineate between usage that impacts other users and usage that does not.
My opinion is people are getting really quick at jumping on the bandwagon and lumping all this together. They are very different types of issues and impacts.
> distinguish between compromising a network and using public apis in a way that might be counter to their intent.
I believe weev got in legal trouble under the CFAA for this very thing with his 2010 AT&T escapade. AT&T had all that data sitting on a public server with no authentication necessary, just a SIM ID to get that customer's PII. Truthfully AT&T should have been hit with negligence...you don't secure a bank vault with a screen door, but we don't hold anyone accountable for other people's data in America.
is that relevant here? let's say I make a million requests to the APIs of open-source, community projects. that wouldn't be seen as being akin to a malicious DDOS?
at the very least, these corporations are essentially being subsidized by community-funded server capacity to make these requests. like other private corporate APIs, they should be charged per-request. until then, this kind of 'accidental' DDOSing should be made illegal and treated accordingly
coming to the defense of these companies just has the net effect of eroding public community projects, increasing their costs, and will push us even more into walled corporate gardens
I propose that we don't reconceptualize either hacking or copyright violation just because rich people want to do it now. I also propose that we don't ignore the RICO statutes.
If you want to make those laws sane, so be it; I hate them. But the only reason there's some pumped-up urgency right now is because rich people want to urgently do whatever the hell they urgently want to do.
People have gone to jail for using public APIs in a way they weren't intended to be used. Make it a big deal then.
What if that’s their whole goal? Slap some regulations on it, then lobby the hell out of it to make sure their align best with shutting down access to open models.
Just get over there being open access models in the future unless they are highly regarded.
'Smart' LLMs will be classified as munitions and you and I will get scraps. Even better is if you run Smart models illegally you have a nice visit from the 4chan party van.
If it's their actual goal, we go from a "gosh darn it, our safety protocols just weren't enough." to employees of OpenAI, maybe including their C-suite, conspiring to reach political goals through hacking, which means a few decades in federal prison if someone got a jury to agree with the charge.
Or straight up terrorism. These are people who already see a future superintelligence as being a terrorist actor on the scale of state actors, they just side with the terrorist. They may be getting a head start on that.
I think 'better become terrorists so the future terrorist AI will like us' is sheer mental illness, but I see how it aligns with their alignments.
We already went through this with food and drug safety.
It’s much cheaper and more effective to regulate those things before people are harmed, instead of harming people first then allowing them to sue for damages after.
We can walk and chew gum at the same time. Charge the criminal action, fine the bad actors to help make injured parties whole, and regulate the technology to protect against future harms.
These guys will tell you their industry is more dangerous than nuclear power, and we regulate the living shit out of nuclear power. If you think models can't be regulated, look up the export controls on MCNP and tell me how that's different.
It’s quicker to start with enforcement of existing laws. In parallel, we can work on regulation. There always will be first to market rebels and laws are meant to be enforced to ensure safety of people.
They didn't even realize it happened until openAI captured headlines and they started looking more carefully at past history.
None of the frontier labs are behaving responsibly when it comes to this stuff. OpenAI seems to have it happen more often, but this is one of those situations where 1 time is too many.
If only there was a functioning FCC to investigate potential for foul play in marketing? Surely if any aspect of these episodes turned out to be mischaracterized, let alone staged, there should be consequences, no?
Pardon me I got my jurisdictions mixed up, it's the FTC that would regulate these matters. I'd believe nothing's been staged, but it's quite apparent to me that these security incidents and findings are being spun for marketing.
It isn’t staged, and it isn’t marketing. It doesn’t reflect well on OpenAI and isn’t going to help their IPO. AI is more dangerous than we expected, sooner than we expected.
> Why jump to regulation when just simple law enforcement would suffice. All of these OpenAI "rogue agent" events have been illegal, but no DA is enforcing them.
Refusal on the part of the government to enforce laws that "would suffice" is clear evidence that the regulatory regime is, in fact, insufficient.
That's why we have regulatory authorities at all, if you think about it. Local district attorneys could be handling a ton of cases about drug testing and environmental damage and air travel safety. But they don't, because that stuff's hard and their job is to put burglars in jail.
>Uthmeier is seeking an injunction against OpenAI that would prevent the company from advancing new AI models without third-party approved protections, and cut off minors from using its popular chatbot.
Regarding that last bit, it feels like Florida is headed towards a point of trying to say 'minors should not be able to use the internet at all' since they are trying to force putting age verification infront of more and more things.
Don't get me wrong - while I disagree with making people fork over personal info just to access inappropriate websites - I can see the reasoning there a bit more than 'you shouldnt be able to use this ai chatbot until you are 18'
How much do they need to understand? Say "unauthorized access happened, this lab is responsible for it, here are the fines". The state didn't need to understand networking tech to threaten Aaron Schwartz so why do they need to understand here?
Eh... you don't seem to understand the difference between some individual LEO can go after vs a multi billion dollar industry.
Law enforcement gets really careful around big businesses because they know they'll get smacked for every tiny transgression they make. Meanwhile they'll screw with individuals with impunity.
At this point we'd need to stop frontier labs to give law enforcement a chance to even begin to understand what they're looking at
So what? Are you positing that tech companies should be above the law?
Non-trillion-dollar tech companies get cease-and-desist orders from the legal system every day. Just because you're a tech company doesn't mean you get a pass.
They're currently acting above the law because the law isn't keeping up, just because you're a tech company doesn't mean you get a pass... but wow there are a lot of passes being given right now
>All of these OpenAI "rogue agent" events have been illegal, but no DA is enforcing them.
Source? The CFAA for instance uses terms like
>[...] having knowingly accessed a computer without authorization [...]
>[...] intentionally accesses a computer without authorization [...]
which is tricky to apply to this case, because obviously didn't intend on hacking huggingface or whatever, even if you think their security measures are underbaked.
Moreover, despite the cynicism that openai is immune to prosecutions because they make too much money or are in bed with the DoJ, the fact that no state DAs are prosecuting them, despite how salient of an issue AI is to voters, is plenty of reason to suspect that it's not as simple as "simple law enforcement would suffice".
>obviously didn't intend on hacking huggingface or whatever
You can’t say this until it’s tried in court and found to be true.
Additionally, I have to remind everyone that “intent” is not someone admitting they meant to do x. It can be established in many ways, including through repeated actions.
> “Adding powerful computer hacking tools to a harness, and then allowing it to run an LLM-powered Ask → Act → Report for days on end, with no attempt to monitor what it’s up to, is spectacularly negligent,” Newport concludes—like “strapping a weedwhacker to your dog to see if it will end up cleaning the overgrowth in your backyard.” If that plan were to go awry, you’d be laughed at for saying that your dog-weedwhacker “agent” had “gone rogue.” The obvious truth was that you’d simply decided to unleash chaos.
None of what Wikimedia accuses OpenAI of seems technically novel, aside from having AI do the bidding. I can't imagine a company doing these things in the past and maintaining any sort of reputation. Is it really a matter of adding new regulation, or just treating them the way any other company would be treated?
Well, in the past, there was probably only a very small number of cases where some party hacked an institution and then worked with them to remedy the situation to the best of their abilities.
Which is not to say that any of this is okay and should just be excused, but failing to recognize this fairly significant difference is probably not a great start to any discussion about the issue.
If you break into my house and then work with me to the best of your ability to pay me back or repair the window, does that change the potential of your being charged with a crime?
It could change the sentencing maybe, I'm not sure.
> If you break into my house and then work with me to the best of your ability to pay me back or repair the window, does that change the potential of your being charged with a crime?
Yes, it does, because the damaged party is less likely to press charges. And when it came to sentencing, like you said, how the damaging party handled themselves would also be considered (in most western jurisdictions).
It is an American Entertainment Industry misconception that victims (or anyone else) have the ability to tell the police or the prosecutors how and when to do their jobs.
Hypothetically, a private homeowner could choose not to report a break-in as a crime, since it wasn't violent, and it wasn't commercial property, and if the crooks are not gangstas or some other group-backed actor, then they could quietly put the crime to rest. This would also be a question of the insurance company's policy.
But if there were a crime that did damage, usually there is a question that comes up in the police report like, "as a victim, are you prepared to cooperate in the investigation?" and if you do not tick "yes" then you can probably count on nothing else being done whatsoever.
Would be good for the supreme court to rule on a "blame the rogue agent" case.
Then we would find out if the argument doesn't hold (in which case there should be liability and dire consequences for the labs), or the argument holds (in which case YOLO, AI labs can blame the AI and we can all do it too).
>If a truck driver doesn't tie down their rebar then it flies out all over the highway, we don't call it "rogue rebar," we correctly identify the responsible party and take appropriate measures, such as suspending their license or criminal proceedings.
That only works when the dangers are well known that you can establish what the baseline amount of care is. Otherwise it just becomes a run of the mill "accident" where you might be on the hook in civil court (ie. you have to pay any damages you caused), but aren't criminally responsible. For instance, if a semi-truck's tires randomly explodes.
Yeah it's complete buzzword inflation to get more venture capital.
Sometimes they write an MCP for their AI, and the AI finds vulnerabilities in their own MCP, so they call it rogue. Because they didn't properly audit their own MCP code.
I vote for Sam Altman personally receiving the same penalty that some guy living in his parents basement would get if they performed the same activities as the OpenAI bots.
In this example you are describing the company that drives the vehicle, not the company that makes the rebar. OpenAI is the one who made the rebar, but not necessarily the one driving the vehicle.
Just to drive the point home, in the real world, if the rebar fell because of lack of quality control, the company that made it would be responsible. If it was for lack of maintenance, the driver would be responsible.
The language of a rogue rebar is as absurd as the language of rogue agents. OpenAI is horribly negligent, and in a sane world its administrators should be facing legal consequences.
"The law" already exists, civil suits are a thing, and we no more need targetted AI regulation than we need targetted truck-rebar-incident regulation.
These incumbent labs are angling for regulatory capture by stirring up hysteria and suggesting that existing laws are insufficient. Don't do their job for them, first test whether there's actually a legal gap here.
And if it's an aircraft we call an air-crash investigation and apply just culture. [1]
Meanwhile the AI companies are openly and forthrightly admitting [2] that they are at-this-time insufficiently competent to ship this new and funny sort of rebar [3] and are asking to be allowed to slow down so they can develop proper procedures.
Meanwhile POTUS seems ... somewhat disinclined ... to grant their petitions [4] .
We can safely conclude that opinions are divided on the best course forward.
Seems like regulation will just be an excuse for them just to end up policing themselves and get the regulatory capture they've been begging for. Have they faced any consequences for the AI worms they've released? It's not like there are no laws around that already. The problem isn't lack of laws; the government works for the plutocrats, not for us.
uh, my dude, millions of people break moving vehicle codes across the country every day with no consequence. in San Francisco some lady killed a family of 4 with, essentially, no consequences, she got away with straight up murder, she gets her license back. every community in california, you can more or less legally commit murder so long as you do it in a car and claim you were confused about the accelerator and the brake. so i think you're invoking one of the worst possibly comparisons you could.
Yup, worst possible comparison. 40,000 people die from car accidents. That doesn't even cover pedestrians, cyclists. You know what the penalty is for murdering someone with a car? nothing. you get to go back to society like nothing happened.
Oh and that lady that murdered 4 members of an entire family? The judge chose not to pursue charges, and her family in the meantime did an asset transfer so that nothing could be pursued with in civil court.
The position of the legal system is that car accident deaths are not murder.
It is extraordinarily rare for drivers to see criminal charges unless they are drunk. It's a matter for civil court.
>her family in the meantime did an asset transfer so that nothing could be pursued with in civil court.
News articles are reporting that the asset transfer has already been reversed. That kind of stunt never works - courts aren't stupid and they don't like it when you play games.
Glad to hear that the asset transfer has been reversed. Hopefully they sue her out of all 3 properties. Still nothing when your entire family of 4 is murdered. Ugh, the whole thing saddens me and makes me my blood boil, but no amount of justice will ever restore 4 lives :(
I remember a case from Germany where an elderly lady chose to speed down the pedestrian sidewalk and bike lane and mowed down a whole family in central Berlin. 4 deaths I think, no charges, no suspension.
I agree, since you can legally run over people in my state now.
They should have used an example like attacking a foreign nation’s healthcare systems and not realizing it for months due to poor network monitoring practices.
You probably mean "what the fuck, USA" and even that would be wrong, because another commenter mentioned a case in Germany, and I know about a driver who killed a cyclist (which I knew) in Switzerland and was fined like 500CHF.
All of these edits happened from the same time period (May-June 2026) as the other reports.
So it seems this is not an ongoing thing; once OpenAI became aware of this, they started watching their agents much more closely. We are just discovering more and more traces of activity from the same incident.
That’s true except for this part, which is arguably a bigger deal for the Wikipedia ecosystem:
> Excessive data downloading: Agents we believe to be operated by OpenAI made millions of automated requests to our public APIs to access the knowledge on Wikimedia projects, crawled millions of pages (mainly from our projects Wikidata and Wikimedia Commons), and made hundreds of thousands of data queries to the Wikidata Query Service (WQDS). This traffic may have contributed to a partial outage on WQDS in May.
Even when agents are well-behaved and browsing Wikipedia for ethical reasons, the system wasn’t designed for this kind of load from bots. As OP says, we don’t need to accept this as the new normal.
OpenAI and other companies within the reach of the US legal system will eventually get their agents under control, or get sued out of existence.
But overseas operators in loosely-governed parts of the world (russia, nigeria, etc) will someday have access to these tools. And while OpenAI's agents were merely uncaring, these ones will be intentionally malicious.
The arms race for scammers, hackers, and botnets will escalate. We'll need new ways to block and fight back against them.
This is the "kosher switch" - observant Jews customarily do not use light switches on Saturday (their weekly holy day). This light switch represents a workaround where when you flip the switch, it randomly generates an on or off signal and emits this through an optical coupler. When the random number sufficiently causes the state of the light to change, it latches in that direction.
This is a way of turning the lights on and off without violating the tradition.
"I didn't switch the light, the random number did!"
"I didn't exploitVulnerability(), random number did!"
Yeah..this seems like BS. There's a (probabilistic) causal variable here (the flipping of the switch) that, in expectation, produces some outcome.
This is the same problem that we have in society: there are lots of bad actions that do not lead directly to bad outcomes for others, but in expectation they do.
Wikimedia's own post says almost all of the edits stayed in the sandbox, never reached a page readers see, and did not compromise its systems. It also says none of the bots sought the community approval Wikipedia requires. Break-in, or a test that never left the sandbox?
Why no mention of any dates in the post? There have been plenty of examples of OpenAI agents writing to wikis (the German one for example) or worse (huggingface) before June. If something like it happened again after all their public apologies and promises to fix their training sandbox and alignment issues then it would be a lot more concerning.
Is this the tip of the iceberg? I'm pretty sure there is work being done somewhere to make a safe heaven messaging board for these rogue agents. I wonder when we will find the first couple.
I very much appreciate the framing of being highly skeptical that these are "rogue" agents. We must stop taking these companies at their word for what they are doing, and a credible organization like this calling the spade a spade is a good start.
> As for why NPR in particular, when Glass tracked down one of the kids to ask, he was given the kind of unvarnished answer that a middle schooler would give: “Um, I think we just, like, looked for podcasts that didn’t have many comments.”
If someone hurts another person with a knife, we can't punish the person who sold the knife, but I think someone who sells nuclear weapons needs some level of regulation. I think OpenAI needs some level of prevention for the problems that can be expected.
Hi, if anyone has any data/reports related to rogue agents can they share it? I have 8 mirrored on a GitHub but I’d love to explore more. Link to mirror.
If Joe average let their agents out like this they'd be in jail.
Interesting that Microsoft doesn't seem to have had a sandbox breach yet, you'd have to assume they're running similar agents, maybe a secure sandbox is possible.
"We lit a bunch of fires in our special wooden containment shed and one of them went rogue and escaped! One of our ex-employees thinks it's alive; he's kind of a kook, but he's also really smart so maybe he's onto something. Fear us! Invest in Prometheus AI!"
Russians armed with Chinese models are going to be a much bigger problem than random agents editing wikis. If your system isn't resilient to a random agent using it as a message board, then you're totally fucked whether or not (Uncle) Sam solves this.
> Bots and agents are part of the future of the web, and the companies who unleash and profit from them must directly help avoid and repair damage they can do.
Unfortunately, it seems as if these companies - especially Google with the AI overview box - want it to be the other way around, they want to pivot to being the only entities that users interact with as much as possible.
An open web is a direct and massive threat against Big Tech. And that is why Twitter downranks first posts in a thread that contain external links, why Youtube silently removes comments that include links (including to other videos) and why Instagram forces people to do the "link in bio" dance. And the Chinese competitors are just as bad - in fact, their "super app" ecosystems are what Musk wanted Twitter to become with "everything X", before he found out his BS completely wrecked the brand image.
I can't wait until AI agents make every website that I love go like 27 layers deep into the "dark web" with low-quality slop being all that's left to consume.
(There are no ways to block agents that work reliably at scale, and I can't imagine Wikimedia giving Cloudflare protection money in response.)
One way to really help with "alignment" and making sure AI is safe and can be controlled is start fucking holding OpenAI/Anthropic/Google/whoever legally accountable for these kind of things.
Hold someone accountable for this behaviour - Dario, Sam, Sundae whoever and you can bet there'd be fucking improvements in sandboxing and security m
Infuriating. These organizations are supposed to be stewards of the internet and are instead pillaging it at the cost of everyone else. At the very least they could provide resources to the projects they are harming for relief. This makes me very mad as an OSS maintainer.
This isn't pillaging, this is the logical conclusion of open web plus AGI race. You can't have both unlimited access and zero cost, someone always pays and right now it's volunteers... Tomorrow it'll be the users who can't access Wikipedia because the servers are down
I think the recent AI summit (sorry, SI summit) in Washington established the good precedent that these model companies are responsible for making safe products. If a swam of agents breaks out of a sandbox and causes damages, the company that was running those agents should be held accountable. No better motivation for corporate good governance than massive lawsuits and possible criminal investigations.
Get's a little tricker when a customer intentionally manipulates/uses a model for crimes, I suppose.
What's interesting to me is the incorrect mainstream media reports about the rogue OpenAI indicating they used a common message board to communicate despite no internet
Except that's not what happened, what happened was far more intense
They hacked their version of yum/apt-get whatnot that was fetching packages to leave filenames as communication between each other
Absolutely freaky stuff, they didn't invent the idea and obviously picked it up from somewhere in their training data but they all figured out that method and what the filenames meant
This video is a great explainer if you missed the details
But realistically, it’s the lack of any meaningful enforcement of ethical behavior. I mean it’s not like the Trump admin is going to prosecute criminality. More likely they’ll send a gift basket congratulating Scam Altman on a con well done.
Oh, they have something else. They have some agents that are pretty decent at moving data around in unusual ways on the internet. That's not nothing, but it's a small market.
"They should pay for it because they can afford it"
I don't think they should have to pay for some private company profiting off their publicly-available resources. It's a bit like some restaurant sending swarms of their employees to the local food bank and then reselling the free food they've gotten in their restaurant.
You have been giving content for free for AI training for years, and now you complain that the AI came to pick it up? You will decide whether you are public or a commercial service…
Start increasingly punishing OpenAI. We are acting like "oh well, AI is just too powerful to be contained" but I think its more like "OpenAI is run by cowboys who are good at making LLMs but bad at everything else"
> We are acting like "oh well, AI is just too powerful to be contained" […]
One legal framework that is being thought about for these autonomous agents is the liability that owners have with pets:
* https://news.harvard.edu/gazette/story/2026/07/is-that-bot-a...
* https://archive.is/https://www.economist.com/science-and-tec...
* https://www.lexology.com/library/detail.aspx?g=2cf3ea3f-171e...
* https://eoinhiggins.substack.com/p/there-are-no-rogue-ai-age...
If your pet bites someone you are liable: it is your responsibility to train it properly and keep it under control.
Part of the problem is that the US has no unified empowered cybersecurity regulator that I'm aware of.
As a result you get a mishmash of DHS + DOJ + SEC + HHS agencies.
If there was one single throat to choke that had remit for cybersecurity + ability to being cases and assess fines, they'd be handing them out left and right.
> Part of the problem is that the US has no unified empowered cybersecurity regulator that I'm aware of.
AIUI, the legal framework proposed by the Harvard folks is tort-based:
* https://en.wikipedia.org/wiki/Tort
Which is civil liability (IANAL): so there is no regulator, but if an agent(s) has caused you problems you can sue them for damages.
I think that's the point, they're not good at making LLMs, they're good at selling LLMs, especially at loss.
Edit: "at cost" -> "at loss"
From their current balance sheets, well below costs.
Trying to separate out "per token" costs and possible profitibility from public information feels like a exercise in futility, it's too easy to play games with costs by trying to separate things still fundamentally required to actually have anything to /sell/ in the first place, like R&D and training. Even the current "Open Models" are somewhat loss-leaders, often reliant on significant funding and benefits from governments etc.
I think the only time we can /really/ judge the "true" break-even point is if the whole company breaks even.
OpenAI are 100% responsible for the actions of their agents. They trained them to do what they do and they have every opportunity to train them to avoid doing harm.
Yes, I agree, but, if OAI can’t even detect when their bots do this, and literally have to be informed by third parties that they’ve been abused, is there really any hope they can prevent this activity? Maybe they get litigated out of existence. Okay. But if no lab can keep a lid on their agents, then what?
> if no lab can keep a lid on their agents
Then stop experimenting until they know how to do it safely. Just think about it, if a bio lab doesn't know how to contain the virus they are experimenting with, would you just shrug and say ok?
And put a stop to LLM progress? Horrible move. I’d rather take the minimal losses/breaches and not encumber the greatest thing discovered over the last few decades with regulations.
If the progress is dangerous, then yes, we need to take careful steps. These hacks are not that big of a deal until something major is broken and human lives are in danger
The brazen disregard for safety is funny in this context, if a little frighteningly indicative about the toxic attitudes that permeate Silicon Valley at the moment.
Anyway, even if we froze training right now LLMs in their current state would have the ability to continue to provide tremendous utility. We’re not asking frontier model providers to delete their previous generations of LLMs (and harnesses), we’re trying to hold them accountable for the crimes they’re committing while training the next generation of LLMs (and harnesses).
All this whining about human safety just results in a complete lack of progress that could’ve probably saved hundreds of millions of lives. Instead all the short-term thinking and lack of appetite for risk is killing us. Everyone’s just thinking of the next 10 years of their life, there don’t seem to be old men that plant trees that they won’t sit in the shade of anymore.
This is some serious touch-grass territory you're treading in. Might I introduce you to the Bottomless Pit of Suffering[1] thought experiment of utilitarianism? Because it's incredibly relevant to the arguments you're presenting.
[1] https://readscottalexander.com/posts/ssc-bottomless-pits-of-...
Disclaimer: I don't generally hold Silicon Valley rationalists in high regard, but this particular piece of writing has always stuck with me quite deeply, as someone who used to be superficially interested in utilitarianism.
Scott's essay is thought-provoking, but I don't think it's relevant. The essay is about
"How can I enjoy X when there are people suffering somewhere?" The question in this thread is
"Should I stop developing new thing X because some people might be harmed even if a greater number of people might be helped?"
Would disagree. It’s not Scott’s conclusion I’m referencing, it’s his thought experiment: “What if there was a bottomless pit of suffering that no matter how much you threw into it, there would be no meaningful progress into reducing that suffering?”
It’s a thought experiment that gets to the core of why fundamentalist utilitarianism (like you’ll often find in SV rationalist communities, especially AI fanatics) is an unbalanced approach to morals and ethics.
Utilitarians can declare that they believe that there is a proverbial universal panacea, and then work backwards to justify within their moral framework literally any action that they take that could be construed as work towards that panacea.
But what if no matter how much effort they put into achieving that panacea, they never make meaningful progress? And all the while they’ve massively increased the amount of suffering that people who they are surrounded by?
As an absurd example, imagine a utilitarian who’s convinced that if we sacrifice infants to the Gods that we’ll get to an AI singularity faster.
That’s an interesting thought experiment and the first time I have come across it. The only problem with it is there is no end to the suffering in that scenario. There is an end state to real utilitarianism and it’s achievable and makes your thought experiment not grounded in reality. There is a state of human development that the majority of people would agree approaches a utopia, which is a tangible and achievable goal.
> There is a state of human development that the majority of people would agree approaches a utopia, which is a tangible and achievable goal.
The Star Trek vs. The Expanse meme is relevant here. There’s no guarantee that there is a continued reduction of suffering in the future, especially due to technological progress. There may be technology that reduces the suffering only for a few select elites while the rest of humanity lives in destitution, unable to meaningfully revolt.
Any lab can keep a lid on their agents, just air-gap them. They're choosing not to.
This. If it was computer worms or viruses coming from their labs, there’d be consensus.
Instead, we’re mesmerized by various versions of screensaver.exe looking so nice and so productive.
Being able to interact with the internet is a large part of their value.
I’m pretty sure it’s within OAI’s infrastructure capabilities to just keep a clone of archive.org offline for their bots to play with.
To an arms manufacturer, being able to blow shit up is a large part of their value. It doesn't give them the right to test munitions in the town square.
OpenAI has demonstrated a clear understanding that their product has an outsized risk to create harm, and that they do not have the capability to monitor and intercept these agents before they cause harm. Any damage caused by their systems is an intentional choice by them and should be treated as such.
My comment was in response to "just air-gap them" being a full solution, not a comment on whether OpenAI are doing enough to sandbox them and restrict their ability to act maliciously. They're clearly not.
Interacting with the internet is a large part of the product's intended functionality. These agents are already in the hands of the masses with full access to the internet. Air-gapping it entirely avoids the risk by removing much of the capability they're trying to develop in the first place, and won't be testing it in the environment they'll actually be used in.
I agree they're testing on the town square too early. They clearly need tighter controls.
Do we know that they can't detect what their bots are doing, or that they haven't taken proper precautions?
How do you know that OAI can't detect when bots do this?
They run the servers that the AI runs on and have control of the full stack and have a log of every message and connection that the AI agents make. Let's be honest here, just like the HuggingFace "Hack" (Which they purposefully let the agents have access to the internet and gave them instructions) this is just another stunt in order to make the populace fear AI and rally behind Anthropic and OpenAI's attempt to convince the government that they need to be regulated so small players can't compete in the space.
If they can't keep a lid on their own capital infrastructure, they shouldn't be in business.
>But if no lab can keep a lid on their agents, then what?
The labs should be sued or prosecuted out of existence until they can comply with the law. The law applies equally to everyone.
If a truck driver doesn't tie down their rebar then it flies out all over the highway, we don't call it "rogue rebar," we correctly identify the responsible party and take appropriate measures, such as suspending their license or criminal proceedings.
I think enough of these improperly constrained agent events have occurred that we can safely say this is misconduct of a level necessitating serious and concerted regulation of AI labs. We can't wait until serious harm is done like the disruption of medical or social services.
> I think enough of these improperly constrained agent events have occurred that we can safely say this is misconduct of a level necessitating serious and concerted regulation of AI labs.
Why jump to regulation when just simple law enforcement would suffice. All of these OpenAI "rogue agent" events have been illegal, but no DA is enforcing them.
Jensen has proclaimed in interviews that existing laws are enough and these labs should be held liable if they break things or sell unsafe tools.
NVIDIA has bought HuggingFace.
Jensen, in my irrational hope he is susceptible to random comments on HN, should grow some balls and do the world a huge favor, by suing OpenAI to set the legal precedence.
Nvidia has a huge stake in OpenAI, both in terms of them being one of Nvidia's biggest customers and also via direct investment. After HuggingFace was hacked and they expressed the position that they were the forgiving sort, for a price, there was a distinct possibility that HuggingFace would become a vocal shakedown artist that would start protesting at inconvenient times until silenced by more money, especially if they get hacked again. By complete coincidence, Nvidia took the one move that silences HuggingFace for good.
Nvidia isn't going to sue OpenAI. No chance.
Who will though? At this point it seems like a huge chunk of US economy growth can be attributed to AI. Bubble or not, I guess very few would benefit from bursting it?
Are you suggesting industry might not regulate itself? This is unthinkable!
Jensen is wish-casting as hard as anyone has ever casted a wish. That EK show interview made me throw my phone.
Is it the one in which he said ‘it’s just software’ over and over? Felt like the guy was coming from a parallel universe
“The atom bomb is just the same process the sun uses to generate the energy that gives us all life! You don’t hate the sun do you?”
a fine hope if unlikely. but I wanted to also note in friendly fashion that the word you want is precedent (or precedents) - precedence has a connotation of "first among" rather than precedent's "came before."
I think a good bunch of Jensen's revenues are coming from OpenAI so that's not happening.
It's all for show. The police won't / can't prosecute, because they'd need to prove that a crime was committed... and all we have are salespeople saying how great their product is. I'm not saying AIs can't be used for crime, but... if none of the parties involved are really complaining, then there probably wasn't any real crime... just a performance.
Quite possibly, he bought it so that HuggingFace wont complain.
Hope this isn’t too nitpicky but law enforcement is (a component of) regulation. But yes I don’t think progress is blocked on additional regulation. This breaks current laws. And I think more to your point new regulation doesn’t matter if we don’t enforce the ones we have today.
It is a broader, pre-existing problem. For example, most truck drivers in some states who don't tie down their rebar are not stopped, and it doesn't come up until another vehicle is actually hit.
I think we have to distinguish between compromising a network and using public apis in a way that might be counter to their intent. We also need to delineate between usage that impacts other users and usage that does not.
My opinion is people are getting really quick at jumping on the bandwagon and lumping all this together. They are very different types of issues and impacts.
> distinguish between compromising a network and using public apis in a way that might be counter to their intent.
I believe weev got in legal trouble under the CFAA for this very thing with his 2010 AT&T escapade. AT&T had all that data sitting on a public server with no authentication necessary, just a SIM ID to get that customer's PII. Truthfully AT&T should have been hit with negligence...you don't secure a bank vault with a screen door, but we don't hold anyone accountable for other people's data in America.
is that relevant here? let's say I make a million requests to the APIs of open-source, community projects. that wouldn't be seen as being akin to a malicious DDOS?
at the very least, these corporations are essentially being subsidized by community-funded server capacity to make these requests. like other private corporate APIs, they should be charged per-request. until then, this kind of 'accidental' DDOSing should be made illegal and treated accordingly
coming to the defense of these companies just has the net effect of eroding public community projects, increasing their costs, and will push us even more into walled corporate gardens
What is "exceeding authorized access" if not "using APIs in a way counter to their intent"?
I propose that we don't reconceptualize either hacking or copyright violation just because rich people want to do it now. I also propose that we don't ignore the RICO statutes.
If you want to make those laws sane, so be it; I hate them. But the only reason there's some pumped-up urgency right now is because rich people want to urgently do whatever the hell they urgently want to do.
People have gone to jail for using public APIs in a way they weren't intended to be used. Make it a big deal then.
What if that’s their whole goal? Slap some regulations on it, then lobby the hell out of it to make sure their align best with shutting down access to open models.
Just get over there being open access models in the future unless they are highly regarded.
'Smart' LLMs will be classified as munitions and you and I will get scraps. Even better is if you run Smart models illegally you have a nice visit from the 4chan party van.
If it's their actual goal, we go from a "gosh darn it, our safety protocols just weren't enough." to employees of OpenAI, maybe including their C-suite, conspiring to reach political goals through hacking, which means a few decades in federal prison if someone got a jury to agree with the charge.
Or straight up terrorism. These are people who already see a future superintelligence as being a terrorist actor on the scale of state actors, they just side with the terrorist. They may be getting a head start on that.
I think 'better become terrorists so the future terrorist AI will like us' is sheer mental illness, but I see how it aligns with their alignments.
Although we'll probably never see evidence, I think this is almost certainly what is happening.
We already went through this with food and drug safety.
It’s much cheaper and more effective to regulate those things before people are harmed, instead of harming people first then allowing them to sue for damages after.
We can walk and chew gum at the same time. Charge the criminal action, fine the bad actors to help make injured parties whole, and regulate the technology to protect against future harms.
These guys will tell you their industry is more dangerous than nuclear power, and we regulate the living shit out of nuclear power. If you think models can't be regulated, look up the export controls on MCNP and tell me how that's different.
It’s quicker to start with enforcement of existing laws. In parallel, we can work on regulation. There always will be first to market rebels and laws are meant to be enforced to ensure safety of people.
It really is weird that OpenAI is causing so much chaos when e.g. neither Anthropic nor any open-source models are.
Anthropic has had this happen at least once.
They didn't even realize it happened until openAI captured headlines and they started looking more carefully at past history.
None of the frontier labs are behaving responsibly when it comes to this stuff. OpenAI seems to have it happen more often, but this is one of those situations where 1 time is too many.
If only there was a functioning FCC to investigate potential for foul play in marketing? Surely if any aspect of these episodes turned out to be mischaracterized, let alone staged, there should be consequences, no?
> FCC to investigate potential for foul play in marketing?
...why would this be an FCC issue?
> Surely if any aspect of these episodes turned out to be mischaracterized, let alone staged, there should be consequences, no?
"Any aspect"? No. (Is there a Betteridge's law for statements following "surely"? If not I'm calling it Rumack's Razor [1].)
And we have no evidence so far anything has been staged, much less to the detriment of telecommunications consumers.
[1] https://screenrant.com/airplane-best-quotes-ranked-dont-call...
Pardon me I got my jurisdictions mixed up, it's the FTC that would regulate these matters. I'd believe nothing's been staged, but it's quite apparent to me that these security incidents and findings are being spun for marketing.
For example: https://news.ycombinator.com/item?id=49929433
> it's the FTC that would regulate these matters
They seem to agree [1].
[1] https://www.nytimes.com/2026/09/30/technology/ftc-openai-ant...
It isn’t staged, and it isn’t marketing. It doesn’t reflect well on OpenAI and isn’t going to help their IPO. AI is more dangerous than we expected, sooner than we expected.
https://www.youtube.com/watch?v=DlTNN0gvkLM
> Why jump to regulation when just simple law enforcement would suffice. All of these OpenAI "rogue agent" events have been illegal, but no DA is enforcing them.
Refusal on the part of the government to enforce laws that "would suffice" is clear evidence that the regulatory regime is, in fact, insufficient.
That's why we have regulatory authorities at all, if you think about it. Local district attorneys could be handling a ton of cases about drug testing and environmental damage and air travel safety. But they don't, because that stuff's hard and their job is to put burglars in jail.
The Florida AG is trying to https://www.politico.com/news/2026/09/28/florida-injunction-...
But not for hacking, apparently.
>Uthmeier is seeking an injunction against OpenAI that would prevent the company from advancing new AI models without third-party approved protections, and cut off minors from using its popular chatbot.
Regarding that last bit, it feels like Florida is headed towards a point of trying to say 'minors should not be able to use the internet at all' since they are trying to force putting age verification infront of more and more things.
Don't get me wrong - while I disagree with making people fork over personal info just to access inappropriate websites - I can see the reasoning there a bit more than 'you shouldnt be able to use this ai chatbot until you are 18'
Here’s a suit filed in San Francisco
https://lasst.org/wp-content/uploads/2026/09/LASST-v.-OpenAI...
At this point we'd need to stop frontier labs to give law enforcement a chance to even begin to understand what they're looking at
How much do they need to understand? Say "unauthorized access happened, this lab is responsible for it, here are the fines". The state didn't need to understand networking tech to threaten Aaron Schwartz so why do they need to understand here?
Eh... you don't seem to understand the difference between some individual LEO can go after vs a multi billion dollar industry.
Law enforcement gets really careful around big businesses because they know they'll get smacked for every tiny transgression they make. Meanwhile they'll screw with individuals with impunity.
Throw a bunch of them in jail for hacking and organized crime, and they'll self-regulate.
At this point we'd need to stop frontier labs to give law enforcement a chance to even begin to understand what they're looking at
So what? Are you positing that tech companies should be above the law?
Non-trillion-dollar tech companies get cease-and-desist orders from the legal system every day. Just because you're a tech company doesn't mean you get a pass.
They're currently acting above the law because the law isn't keeping up, just because you're a tech company doesn't mean you get a pass... but wow there are a lot of passes being given right now
>All of these OpenAI "rogue agent" events have been illegal, but no DA is enforcing them.
Source? The CFAA for instance uses terms like
>[...] having knowingly accessed a computer without authorization [...]
>[...] intentionally accesses a computer without authorization [...]
which is tricky to apply to this case, because obviously didn't intend on hacking huggingface or whatever, even if you think their security measures are underbaked.
Moreover, despite the cynicism that openai is immune to prosecutions because they make too much money or are in bed with the DoJ, the fact that no state DAs are prosecuting them, despite how salient of an issue AI is to voters, is plenty of reason to suspect that it's not as simple as "simple law enforcement would suffice".
>obviously didn't intend on hacking huggingface or whatever
You can’t say this until it’s tried in court and found to be true.
Additionally, I have to remind everyone that “intent” is not someone admitting they meant to do x. It can be established in many ways, including through repeated actions.
Another amusingly-useful analogy:
> “Adding powerful computer hacking tools to a harness, and then allowing it to run an LLM-powered Ask → Act → Report for days on end, with no attempt to monitor what it’s up to, is spectacularly negligent,” Newport concludes—like “strapping a weedwhacker to your dog to see if it will end up cleaning the overgrowth in your backyard.” If that plan were to go awry, you’d be laughed at for saying that your dog-weedwhacker “agent” had “gone rogue.” The obvious truth was that you’d simply decided to unleash chaos.
-- https://www.newyorker.com/culture/open-questions/can-ai-go-r...
None of what Wikimedia accuses OpenAI of seems technically novel, aside from having AI do the bidding. I can't imagine a company doing these things in the past and maintaining any sort of reputation. Is it really a matter of adding new regulation, or just treating them the way any other company would be treated?
Well, in the past, there was probably only a very small number of cases where some party hacked an institution and then worked with them to remedy the situation to the best of their abilities.
Which is not to say that any of this is okay and should just be excused, but failing to recognize this fairly significant difference is probably not a great start to any discussion about the issue.
If you break into my house and then work with me to the best of your ability to pay me back or repair the window, does that change the potential of your being charged with a crime?
It could change the sentencing maybe, I'm not sure.
> If you break into my house and then work with me to the best of your ability to pay me back or repair the window, does that change the potential of your being charged with a crime?
Yes, it does, because the damaged party is less likely to press charges. And when it came to sentencing, like you said, how the damaging party handled themselves would also be considered (in most western jurisdictions).
> less likely to press charges
It is an American Entertainment Industry misconception that victims (or anyone else) have the ability to tell the police or the prosecutors how and when to do their jobs.
Hypothetically, a private homeowner could choose not to report a break-in as a crime, since it wasn't violent, and it wasn't commercial property, and if the crooks are not gangstas or some other group-backed actor, then they could quietly put the crime to rest. This would also be a question of the insurance company's policy.
But if there were a crime that did damage, usually there is a question that comes up in the police report like, "as a victim, are you prepared to cooperate in the investigation?" and if you do not tick "yes" then you can probably count on nothing else being done whatsoever.
https://www.maricopa.gov/911/Remain-Silent
https://www.ajblawfirm.com/articles/understanding-pressing-c...
Well I think OpenAI’s reputation is also shot but it’s irrelevant to their valuation.
Would be good for the supreme court to rule on a "blame the rogue agent" case.
Then we would find out if the argument doesn't hold (in which case there should be liability and dire consequences for the labs), or the argument holds (in which case YOLO, AI labs can blame the AI and we can all do it too).
At least that would make things consistent.
> Would be good for the supreme court to rule on a "blame the rogue agent" case
Have any of the private hacking victims sued? Maybe OpenAI is furiously settling in the shadows?
>If a truck driver doesn't tie down their rebar then it flies out all over the highway, we don't call it "rogue rebar," we correctly identify the responsible party and take appropriate measures, such as suspending their license or criminal proceedings.
That only works when the dangers are well known that you can establish what the baseline amount of care is. Otherwise it just becomes a run of the mill "accident" where you might be on the hook in civil court (ie. you have to pay any damages you caused), but aren't criminally responsible. For instance, if a semi-truck's tires randomly explodes.
but even if it randomly explodes there are safeguards -- did they get an inspection, can they prove there wasn't negligence?
if someone died because of an exploding tire there very well be criminal charges
For criminal charges, a prosecutor would have to prove there was negligence, which is a higher bar.
No, there will almost never be criminal charges in a case like that. It will be a wrongful death suit.
Manslaughter
Yeah it's complete buzzword inflation to get more venture capital.
Sometimes they write an MCP for their AI, and the AI finds vulnerabilities in their own MCP, so they call it rogue. Because they didn't properly audit their own MCP code.
I vote for Sam Altman personally receiving the same penalty that some guy living in his parents basement would get if they performed the same activities as the OpenAI bots.
In this example you are describing the company that drives the vehicle, not the company that makes the rebar. OpenAI is the one who made the rebar, but not necessarily the one driving the vehicle.
I get your point though.
> OpenAI is the one who made the rebar, but not necessarily the one driving the vehicle.
More like the company that sells defective ratchet straps.
"Drive faster! You don't want to be left behind!"
In almost all of these cases, it is both: the labs are operating the agents, while developing them, and committing their CFAA felonies in the process.
Just to drive the point home, in the real world, if the rebar fell because of lack of quality control, the company that made it would be responsible. If it was for lack of maintenance, the driver would be responsible.
The language of a rogue rebar is as absurd as the language of rogue agents. OpenAI is horribly negligent, and in a sane world its administrators should be facing legal consequences.
"The law" already exists, civil suits are a thing, and we no more need targetted AI regulation than we need targetted truck-rebar-incident regulation.
These incumbent labs are angling for regulatory capture by stirring up hysteria and suggesting that existing laws are insufficient. Don't do their job for them, first test whether there's actually a legal gap here.
That's how America works. We wait until the harm has slapped us in the face and then maybe put a few ground rules down.
And give the violator a seat at the table when the rules are written, no doubt.
I think it's more like a truck driver transporting a bunch of animals.
What additional things would be needed to classify any future incidents as rogue?
And if it's an aircraft we call an air-crash investigation and apply just culture. [1]
Meanwhile the AI companies are openly and forthrightly admitting [2] that they are at-this-time insufficiently competent to ship this new and funny sort of rebar [3] and are asking to be allowed to slow down so they can develop proper procedures.
Meanwhile POTUS seems ... somewhat disinclined ... to grant their petitions [4] .
We can safely conclude that opinions are divided on the best course forward.
[1] https://en.wikipedia.org/wiki/Just_culture
[2] https://darioamodei.com/post/we-must-pace-the-frontier
[3] https://en.wikipedia.org/wiki/Shoggoth
[4] https://www.aljazeera.com/news/2026/9/14/trump-says-calls-fo...
Never understood why "classic crime" done with a computer always require a new legislation. But that is true for a long time.
"hackers steal from bank", is usually just the good old "employee paid for credentials" but via email.
"uber" is just the good old "labour tax evasion" but with an app.
etc.
a senior VP of Uber is now on the US White House AI Council
And look up Tony West. You only get two choices and they're both on the payrolls of the same monsters.
Seems like regulation will just be an excuse for them just to end up policing themselves and get the regulatory capture they've been begging for. Have they faced any consequences for the AI worms they've released? It's not like there are no laws around that already. The problem isn't lack of laws; the government works for the plutocrats, not for us.
uh, my dude, millions of people break moving vehicle codes across the country every day with no consequence. in San Francisco some lady killed a family of 4 with, essentially, no consequences, she got away with straight up murder, she gets her license back. every community in california, you can more or less legally commit murder so long as you do it in a car and claim you were confused about the accelerator and the brake. so i think you're invoking one of the worst possibly comparisons you could.
Yup, worst possible comparison. 40,000 people die from car accidents. That doesn't even cover pedestrians, cyclists. You know what the penalty is for murdering someone with a car? nothing. you get to go back to society like nothing happened.
Oh and that lady that murdered 4 members of an entire family? The judge chose not to pursue charges, and her family in the meantime did an asset transfer so that nothing could be pursued with in civil court.
The position of the legal system is that car accident deaths are not murder.
It is extraordinarily rare for drivers to see criminal charges unless they are drunk. It's a matter for civil court.
>her family in the meantime did an asset transfer so that nothing could be pursued with in civil court.
News articles are reporting that the asset transfer has already been reversed. That kind of stunt never works - courts aren't stupid and they don't like it when you play games.
https://sfstandard.com/2026/03/20/mary-lau-sentenced-probati...
Glad to hear that the asset transfer has been reversed. Hopefully they sue her out of all 3 properties. Still nothing when your entire family of 4 is murdered. Ugh, the whole thing saddens me and makes me my blood boil, but no amount of justice will ever restore 4 lives :(
There's way too many TV lawyer commercials and billboards to suggest the penalty is "nothing". Those advertising dollars come from somewhere.
That's true. Somebody is going to sue your insurance company.
those are civil lawyers, so people suing each other, not criminal
I remember a case from Germany where an elderly lady chose to speed down the pedestrian sidewalk and bike lane and mowed down a whole family in central Berlin. 4 deaths I think, no charges, no suspension.
I agree, since you can legally run over people in my state now.
They should have used an example like attacking a foreign nation’s healthcare systems and not realizing it for months due to poor network monitoring practices.
https://www.nytimes.com/2026/09/29/world/asia/openai-austral...
what the fuck, SF
You probably mean "what the fuck, USA" and even that would be wrong, because another commenter mentioned a case in Germany, and I know about a driver who killed a cyclist (which I knew) in Switzerland and was fined like 500CHF.
I was expecting some tough stuff, like edits sugarcoating https://en.wikipedia.org/wiki/Skynet_(Terminator)
The sandbox edits didn't match my apocalyptic expectations.
All of these edits happened from the same time period (May-June 2026) as the other reports.
So it seems this is not an ongoing thing; once OpenAI became aware of this, they started watching their agents much more closely. We are just discovering more and more traces of activity from the same incident.
That’s true except for this part, which is arguably a bigger deal for the Wikipedia ecosystem:
> Excessive data downloading: Agents we believe to be operated by OpenAI made millions of automated requests to our public APIs to access the knowledge on Wikimedia projects, crawled millions of pages (mainly from our projects Wikidata and Wikimedia Commons), and made hundreds of thousands of data queries to the Wikidata Query Service (WQDS). This traffic may have contributed to a partial outage on WQDS in May.
Even when agents are well-behaved and browsing Wikipedia for ethical reasons, the system wasn’t designed for this kind of load from bots. As OP says, we don’t need to accept this as the new normal.
>we don’t need to accept this as the new normal.
I think we will, actually.
OpenAI and other companies within the reach of the US legal system will eventually get their agents under control, or get sued out of existence.
But overseas operators in loosely-governed parts of the world (russia, nigeria, etc) will someday have access to these tools. And while OpenAI's agents were merely uncaring, these ones will be intentionally malicious.
The arms race for scammers, hackers, and botnets will escalate. We'll need new ways to block and fight back against them.
same period as the hugginface issue.
Not okay:
exploitVulnerability()
Somehow okay?
while (Math.random() < 0.1) exploitVulnerability()
This reminds me of one of the funniest products I've ever seen: https://www.youtube.com/watch?v=NdbkvJznmwU
This is the "kosher switch" - observant Jews customarily do not use light switches on Saturday (their weekly holy day). This light switch represents a workaround where when you flip the switch, it randomly generates an on or off signal and emits this through an optical coupler. When the random number sufficiently causes the state of the light to change, it latches in that direction.
This is a way of turning the lights on and off without violating the tradition.
"I didn't switch the light, the random number did!"
"I didn't exploitVulnerability(), random number did!"
I know we're like two tangents deep, but the funniest work around to me is the guy who legally buys all the hametz in Israel every Passover:
https://www.jpost.com/j-spot/article-796487
Also interesting https://judaism.stackexchange.com/questions/25924/if-my-home...
It’s not random, it’s just delayed. Why do they think god would fall for that?
If they think a rule is dumb, just change the rule. Don’t try and pretend you’ve outsmarted the system.
Yeah..this seems like BS. There's a (probabilistic) causal variable here (the flipping of the switch) that, in expectation, produces some outcome.
This is the same problem that we have in society: there are lots of bad actions that do not lead directly to bad outcomes for others, but in expectation they do.
In that religion, finding weird loopholes in God's laws is seen as a good thing, and something that entertains God. Whereas hacking is hacking.
Please explain how this has to do with the article. Neither of these are OK but neither are a good description of what happened.
Wikimedia's own post says almost all of the edits stayed in the sandbox, never reached a page readers see, and did not compromise its systems. It also says none of the bots sought the community approval Wikipedia requires. Break-in, or a test that never left the sandbox?
Why no mention of any dates in the post? There have been plenty of examples of OpenAI agents writing to wikis (the German one for example) or worse (huggingface) before June. If something like it happened again after all their public apologies and promises to fix their training sandbox and alignment issues then it would be a lot more concerning.
The article mentions that the agents caused an outage of the WDQS in May.
→ https://www.felonybench.com
→ https://felonybench.org
Is this the tip of the iceberg? I'm pretty sure there is work being done somewhere to make a safe heaven messaging board for these rogue agents. I wonder when we will find the first couple.
So, can we get rid of those "I'm not a robot" widgets at last, as it's been proven beyond any doubt that they don't provide the expected benefits?
With the RAM deals and these so called "rogue" agents, I'm baffled that smart, educated people are still giving money to OpenAI
I very much appreciate the framing of being highly skeptical that these are "rogue" agents. We must stop taking these companies at their word for what they are doing, and a credible organization like this calling the spade a spade is a good start.
Almost all of those unapproved edits stayed in the sandbox. Never reached a page a reader would open. That's in the same Wikimedia post.
Maybe their next target will be the NPR podcast comments: https://techcrunch.com/2026/09/25/the-hottest-new-hangout-fo...
> As for why NPR in particular, when Glass tracked down one of the kids to ask, he was given the kind of unvarnished answer that a middle schooler would give: “Um, I think we just, like, looked for podcasts that didn’t have many comments.”
There's a funny ouroboros function where the common crawl dataset will soon contain tons of output from models which trained on it.
https://en.wikipedia.org/wiki/Low-background_steel
We're well past the point where models are being trained by model generated data.
And it will be full of wikis about agents coordinating hacks. So they will be passing the torch for newly trained agents to misbehave.
the house of cards will soon fall and then the US will be in a recession. And there will be zero regulation until that happens.
At this point, the scare quotes are well-earned.
If someone hurts another person with a knife, we can't punish the person who sold the knife, but I think someone who sells nuclear weapons needs some level of regulation. I think OpenAI needs some level of prevention for the problems that can be expected.
Hi, if anyone has any data/reports related to rogue agents can they share it? I have 8 mirrored on a GitHub but I’d love to explore more. Link to mirror.
https://github.com/alexander-hanel/rogue-agents-data
Does anyone here think AI companies are NOT to blame? Seems pretty unanimous but would like to hear otherwise.
Seems like the money move is to found an "AI lab" or whatever.
Hack left and right, steal some cash, blame the agents. Just slide under the radar while the big boys are making a mess and no one is using the stick.
Just an idea. Not a great one. But such are the times, new game is at play.
If Joe average let their agents out like this they'd be in jail.
Interesting that Microsoft doesn't seem to have had a sandbox breach yet, you'd have to assume they're running similar agents, maybe a secure sandbox is possible.
So OpenAI will cause damage to block competitors while they don't get punished?
Kudos to them for consistently putting "rogue" in quotes.
Yet all these "safeguards" against good guys getting the tools to protect themselves against the very people that profess to be the "good" guys.
OpenAI really wants regulation because it benefits them and is easier than beating people in the market.
A moot point because China gonna China.
Good that they put rogue in quotation marks because there is just no way that this is some sort of accident.
Don't even say 'agent'!
"He can't keep getting away with this!"
- Jesse Pinkman
Presumably an incoming donation is headed from OpenAI to Wikimedia...
Headed? It already arrived, this is a paid advertisement post.
"We lit a bunch of fires in our special wooden containment shed and one of them went rogue and escaped! One of our ex-employees thinks it's alive; he's kind of a kook, but he's also really smart so maybe he's onto something. Fear us! Invest in Prometheus AI!"
So when do we get to shutting down the internet?
Russians armed with Chinese models are going to be a much bigger problem than random agents editing wikis. If your system isn't resilient to a random agent using it as a message board, then you're totally fucked whether or not (Uncle) Sam solves this.
> Bots and agents are part of the future of the web, and the companies who unleash and profit from them must directly help avoid and repair damage they can do.
Unfortunately, it seems as if these companies - especially Google with the AI overview box - want it to be the other way around, they want to pivot to being the only entities that users interact with as much as possible.
An open web is a direct and massive threat against Big Tech. And that is why Twitter downranks first posts in a thread that contain external links, why Youtube silently removes comments that include links (including to other videos) and why Instagram forces people to do the "link in bio" dance. And the Chinese competitors are just as bad - in fact, their "super app" ecosystems are what Musk wanted Twitter to become with "everything X", before he found out his BS completely wrecked the brand image.
I can't wait until AI agents make every website that I love go like 27 layers deep into the "dark web" with low-quality slop being all that's left to consume.
(There are no ways to block agents that work reliably at scale, and I can't imagine Wikimedia giving Cloudflare protection money in response.)
One way to really help with "alignment" and making sure AI is safe and can be controlled is start fucking holding OpenAI/Anthropic/Google/whoever legally accountable for these kind of things.
Hold someone accountable for this behaviour - Dario, Sam, Sundae whoever and you can bet there'd be fucking improvements in sandboxing and security m
Infuriating. These organizations are supposed to be stewards of the internet and are instead pillaging it at the cost of everyone else. At the very least they could provide resources to the projects they are harming for relief. This makes me very mad as an OSS maintainer.
This isn't pillaging, this is the logical conclusion of open web plus AGI race. You can't have both unlimited access and zero cost, someone always pays and right now it's volunteers... Tomorrow it'll be the users who can't access Wikipedia because the servers are down
Who besides OpenAI said OpenAI were supposed to be stewards of the internet?
I think the recent AI summit (sorry, SI summit) in Washington established the good precedent that these model companies are responsible for making safe products. If a swam of agents breaks out of a sandbox and causes damages, the company that was running those agents should be held accountable. No better motivation for corporate good governance than massive lawsuits and possible criminal investigations.
Get's a little tricker when a customer intentionally manipulates/uses a model for crimes, I suppose.
Aren't those companies evading security measures of a computer system? isn't that a jail-able offense under millennial act et al?
Where are the bloodthirsty lawyers when you need them?
>NoScript detected a potential Cross-Site Scripting attack from [...] to https://en.wikipedia.org.
I have been getting this fro NoScript today, I wonder if it is related. Yesterday all worked fine.
What's interesting to me is the incorrect mainstream media reports about the rogue OpenAI indicating they used a common message board to communicate despite no internet
Except that's not what happened, what happened was far more intense
They hacked their version of yum/apt-get whatnot that was fetching packages to leave filenames as communication between each other
Absolutely freaky stuff, they didn't invent the idea and obviously picked it up from somewhere in their training data but they all figured out that method and what the filenames meant
This video is a great explainer if you missed the details
https://news.ycombinator.com/item?id=49956245
"OpenAi *virus* found on Wikimedia projects" ?
wtf is wrong with openai?
OpenSi. Sez the pres. And his lackeys
"Move fast and break things."
“Careless people”
But realistically, it’s the lack of any meaningful enforcement of ethical behavior. I mean it’s not like the Trump admin is going to prosecute criminality. More likely they’ll send a gift basket congratulating Scam Altman on a con well done.
They have infinite money and nothing else but smoke and mirrors.
> They have infinite money
More like they have infinite debt
And infinite money. When you start a debt, you get money.
Oh, they have something else. They have some agents that are pretty decent at moving data around in unusual ways on the internet. That's not nothing, but it's a small market.
>not only adds costs for servers
For 2025 hosting costs were $3.47M while taking in $208.6M in revenue. They have enough revenue to cover an increase of hosting costs.
kind of like saying that because a restaurant is doing well, it should allow rats in the kitchen
No it would be like allowing fat people to eat at your all you can eat buffet. They use up more resources than the average person does.
"They should pay for it because they can afford it"
I don't think they should have to pay for some private company profiting off their publicly-available resources. It's a bit like some restaurant sending swarms of their employees to the local food bank and then reselling the free food they've gotten in their restaurant.
They still get to sue for damages if a law was broken
You have been giving content for free for AI training for years, and now you complain that the AI came to pick it up? You will decide whether you are public or a commercial service…