I have some sympathy for Anthropic here because I've seen the headlines after OpenAI failed to report a shooter in a similar situation. So from their perspective, it's damned-if-you-don't, damned-if-you-do.
However, people need to get it in their heads that they're not chatting with their secret BFF, they're chatting with Big Tech. Before LLMs, Big Tech had no way to scrutinize the bulk of what was going on within their services, so you could have a secret hate diary in Google Docs. Now, everything you say or write can be automatically screened for red flags on a planetary scale, and probably will be because that's what the regulators and "concerned citizens" will demand. In a couple of years, you'll be biting your tongue a lot more often in private chats.
Not just bad headlines; OpenAI is actively being sued [1] for this:
OpenAI’s stated rationale was a concern for the shooter’s privacy, but its own interests
better explain its silence. Upon information and belief, OpenAI was seeking to avoid implementation of a
hard line rule to refer planning of real-world violence to authorities, perhaps due to how frequently its
product is implicated in threats to human life. Requiring such disclosures would be incompatible with the
company’s public position that ChatGPT is safe. It could also threaten the valuation underlying OpenAI’s
anticipated initial public offering. Rather than expose those risks, OpenAI accepted the consequences of
its silence. A mass murder in the only secondary school in Tumbler Ridge followed.
Accordingly, the Crown, led by Attorney General Sharma, and SD59 jointly bring this
action to hold OpenAI and Sam Altman accountable for designing a dangerous product, distributing it to
every home with internet access, ignoring the warnings of their own safety team,
refusing to notify authorities when they knew the shooter was planning gun violence, inviting the shooter back onto the
platform after deactivating the shooter’s account, and choosing corporate self-interest over the lives of
children. They seek compensation for the not just foreseeable but known harm OpenAI inflicted, the
damages that they incurred and are incurring, and injunctive relief to ensure that this tragedy does not
happen again.
That's what the suit alleges. Also that it had been automatically flagged to an internal human review team who had concluded that there was a credible, specific risk of harm to others and recommended notifying police, and that it was OpenAI leadership that rejected that recommendation.
You can see it all in GP's link. It's very readable.
So mentioning killing in passing is fine, but planning with the tool is not? How should LLM providers draw the line? I'd prefer if they didn't scrutinize any of my inputs to see if I am committing a felony or not in the first place - but that ship has sailed.
They draw the line using a team of humans who evaluate whether the threat is real and give a recommendation to management.
If you operate a gun store and someone reveals to you that they intend to use your product to shoot up a school you have a similar moral obligation, I think. This isn't a special case.
You're really stretching here for some reason I can't fathom, but yes - if you write a credible threat to shoot up your school on a crumpled up paper and someone finds it, they should report it. I'm not sure that complicating this with which kind of app it's written in is all that useful ...
Is this a point you're arguing seriously? If you find someone's plans to, for example, shoot up a school crumpled in a note on the ground that should be...ignored? Or what? I struggle to comprehend what you're actually arguing for here.
I think I pretty much agree with you that if someone finds evidence of danger to someone else's life then it is a fair expectation that they do something with that evidence. And despite that I still believe that is not a good enough reason to eliminate someone's right to privacy, and yet still that right doesn't mean that we should be able to demand privacy in all products. I think it is fairly clear that these big LLMs are not private products and we shouldn't expect them to be, but I'm still in support of offering private alternatives that people could use without having every input scrutinized.
I'm still puzzled that people's default assumption isn't that somebody is reading all of their internet communications. Ever since the Snowden revelations of 2013 I've more or less assumed that everything I type into a computer is stored in a government database somewhere. Not that I actually believe it is 100% of the time; there's a spectrum of trust, so I'm more confident that local apps on my Linux desktop are secure, somewhat confident in the end-to-end encryption of certain apps on my iPhone, but all bets are off for non-E2E encrypted data going across the internet.
People not reading it is unfortunately pretty close to the truth.
You can bet that everything sent across the Internet is stored somewhere. But read? Yeah right. People don't seems to realize that there are 6B+ people online and just how big a number 6B is. If you have a staff of 30-40K people (like the FBI or NSA) and they spend 40 hours/week doing nothing but reading Internet posts and the average person spends 4 hours/week writing Internet posts (likely a huge underestimate) and the FBI agent can read 5x as fast as the person can post, then the collective manual surveillance capacity is about 2M people. That gives a 1 in 3000 chance that a random Internet user would be surveilled (though it is decidedly not random). With more realistic numbers of maybe 5000 analysts reviewing potential threats, spending 10 hours/week on it (so they have time to actually follow up on threats, attend meetings, communicate with their boss & coworkers, etc.), and the average person spending more like 25 hours/week posting on the Internet, that's a surveillance capacity of 10K people and your chances are more like 1 in 600K.
AI will give you only what you prompt it for, and the prompts are still designed by human analysts. What's changed with it is that now the 10K people surveilled will be anyone that "fits" in a category that the government deems criminal (say being transgender, or foreign, or left-wing, or criticizing ICE) instead of actual criminals. It shifts targeting, not capacity. If you wanted to be an actual criminal and are willing to do it in creative ways that bear no resemblance to other major categories of criminality (or lets be real, political disfavor), there is no time better than the present.
Sorry mate. The one thing llms effectively guaranteed is that short of wonks like me, no one is actually reading the source. Everything is a summary of summary of a summary.
FWIW, I feel you; I too try to leave internet a mildly more amusing place to be.
People don’t want to believe the Stasi is monitoring them. They want to believe the world is a nice place just like where they grew up.
Also, unlike the bad old days when 1 in 3 was an informant, now ordinary people aren’t in “informant loop” of providing information on others, so they aren’t thinking about being informed on either.
Our devices, even Anthropic in this case, set an expectation that doesn't match reality. The software and devices we use are marketed and sold to us to be used for our private documents, photos, and conversations. I can't blame people for using them for their intended purpose. They shouldn't assume that someone is carefully watching everything they do and judging their every word and action looking for evidence of crimes. If we want people to think of the devices and services they bought and use as belonging to someone else and without any expectation for privacy we should really start with forcing companies to change how they market their products and services to reflect that reality.
> I'm more confident that local apps on my Linux desktop are secure
Why should you be? Linux gives us software we can reasonably say we own, but not hardware. Everything you type into your local linux apps can be being monitored by your processor and whatever is sitting in the CSME/PSP subsystem which you don't have permission to access, but which is always running even when your computer is off. We need fully open, documented, auditable hardware if allowing any third party access to our devices means our private documents will make us all into criminal suspects.
Most of the hardware isn't made in the US, so either the spooks would need agreements with every offshore business, intercept every hardware shipment, or convince every vendor operating in the US, even if they are not locally owned, to do this for all hardware sold here, and convince all of these offshore vendors not to ever leak the fact that it is happening.
On top of that, the data has to pass through several layers of routers and access points we control, which makes at least the fact of exfiltration visible. Plenty of people have a very strong vested interest in preventing data exfiltration from company hardware and so the government would have to somehow get in touch with every one of them to tell them to stay quiet before they leak any hints of it happening.
It's possible for targeted shipments, but the prospect of it being done for all consumer hardware sold in the US is a bit far fetched.
We know for a fact the government does intercept hardware shipments already on some scale. It's a lot easier when there are only a few chip makers being used for most devices. Intel and AMD alone cover the vast majority of the CPU market for desktops and laptops. It's basically the same situation for the wireless chipsets in every mobile device. No need to worry about dealing with about every cell phone manufacturer directly when you only need to hit up the extremely small number of companies every manufacturer has to get their chips from.
> Plenty of people have a very strong vested interest in preventing data exfiltration from company hardware and so the government would have to somehow get in touch with every one of them to tell them to stay quiet before they leak any hints of it happening.
Whistleblowers are extraordinarily rare. Edward Snowden worked with many many others. Any of them could have come forward at any time, but they didn't. Only one AT&T employee came forward to tell the public about Room 641A (https://en.wikipedia.org/wiki/Room_641A). The government marched into AT&T's building, took over part of their offices, rerouted their network and the whole AT&T backbone into their offices and through hardware. You can bet that more than one person noticed that. Government has no problems at all with going to a business with guns and gag orders and telling people to keep their mouths shut. Companies have no problems keeping quiet about what's happening either.
No one who isn't directly in the know is going to notice if someone's CPU (which is running it's own network stack) sends a few extra encrypted packets going to the servers of major internet companies (microsoft, cloudflare, apple, google) to either be collected at those end points or just picked up as it passes across the internet backbone.
Maybe if we had a ton more competition in things like chip makers, ISPs, operating systems, etc. it might be more difficult, but the way things are it'd be easy.
Yes, I mentioned the Tailored Access Operations NSA unit in a sibling comment. The thing is, open source hardware won't save you from any of this unless you have the manufacturing skills and equipment to build it yourself. Just like with open source software, if you aren't building it yourself, with your own hands, you can't be sure that what end result you have actually matches the code that you've read, unless there's a published checksum that you've also verified is real. Quite a bit harder to do with hardware than software though, especially since every single piece of the stack, including monitor cables, have been targeted by TAO.
Companies like Google and Microsoft pay top dollar to build special workstations to interact with production systems. These are not just special because of the software they run, but also the chips these machines are built with.
Sure, I'll take your comment at face value, not all consumer hardware has spyware, but it cannot be ruled out now, in the future, or at least now for targeted high value customers.
That’s… really not true or at least not as dramatic as that statement makes it sound.
Those “special workstations” are mostly about lack of features they view as potential attack vectors (external or internal motherboard ports, replaceable/non-soldered components, thunderbolt, anything extensible or “repairable”) and mandate certain hardware features (hardware tpm, locked secure boot, locked UEFI). Other than that, those workstations are Lenovo, Samsung, or Dell machines running typical chips. They may pick certain CPU models or chips but that’s mostly about avoiding new or “cool” features that may not have matured enough yet. Most of them are moving to thin clients though. Where you use that “special workstation” to remote into an VM that can access production systems. And that part is partially about controlling the software running in the VM and making sure updates can be forced “offline” on the VM even if you haven’t connected to it in few weeks.
For these kinds of operations, open source hardware won't save you unless you build it yourself, much like open source software. At the end of the day, you're hoping that the pre built thing matches what it claims to be.
The hardware case of TAO is still targeted and didn't rely on the manufacturer's compliance and secrecy for their entire line of products, though. The scale of that to my mind is quite the qualitative difference.
It's true that you'd have a hard time knowing a chip wasn't swapped someone on its way to you. There are a few things that would help but nothing easy. Getting the hardware into retail locations so that anyone could walk in off the street and buy a system with cash. Random spot checks, chip by chip. Packing the system in colored beads/rice to prevent tampering after it's in a box.
At this point, even if I could never be sure my system was secure, just knowing that systems were purchased, tested, and verified would make me feel a lot better. It'd be nice to have confirmation that it's even possible to make and sell a system that isn't backdoored in this country.
> I'm still puzzled that people's default assumption isn't that somebody is reading all of their internet communications.
I'm also puzzled by the phenomenon of using ones real name on the Internet (outside a professional context). I believe this began to occur around the time facebook became popular.
I've always used my real name, since I first got on to the internet in the early 90s. It's never been a problem. Big Tech and your own government know who you really are anyway, as do the big online advertisers. Pseudonyms are no defence against that. There is a potential risk from griefers, but they have ways and means as well.
That's definitely not the good word to use. It is most likely made with the intent to be privacy-friendly, but they are unfortunately anything but secure (including the whole Linux userspace), and especially in the age of agents it would be the best if everyone understood it that you are a single bash/npm install/malicious PDF away from everything bad happening with your data. But rest assured, your video driver won't get updated!
For technical people, this is incredibly old news.
For non-technical people, it isn't really news, because they already forgot about it after reading it. Maybe they'll be a little more monitored in their own typing for like... a day or two.
One of the hardest lessons to internalize, and keep internalized, as someone who works on and writes software, is the vast, vast, vast majority of the Public doesn't understand even the most basic shit about software. It just does stuff. Hopefully the stuff is good. That's it, beginning, middle, and end.
"Why would you think x would y" is a poor framing. They didn't think about x or y because they don't care. The phone works, that's the beginning and end of their interest in the subject.
You're completely right. I've internalized this on the technical side, but I still find it puzzling politically. Surveillance has been a salient issue in U.S. politics for almost 100 years now, since the invention of the telephone. I would like to think that most people are at least vaguely aware of Watergate, the Patriot Act, and Snowden. Then again, I should probably stop assuming that people know basic history, given the current state of education.
Honestly the older I get and the more I learn about criminals, of all stripes really be they petty thieves all the way up to state actors behind the most atrocious crimes against humanity we know of, they are all a bit on the dumb side. The ones that get caught are, anyway. Like so very often the perpetrator of a given crime just did one incredibly bone-headed thing that brought the cops directly to their door.
I think in part it's selection bias? Like if you're smart enough to get by honestly, you're probably also smart enough to realize getting by honestly is just a way more comfortable way to live. The only reason you'd probably cross that line is because your principals, whatever they may be, conflict with those laws, or your life circumstances are so bad that you have no choice BUT to turn to crime.
And that cuts the other way too: if you're dumb enough to think you'd NEVER get caught for a burglary, for example, you'd probably be way more down to plan and execute one, failing to consider that most thieves aren't caught when they steal the shit, they're caught when they try and sell it later.
Those in power are often dumb criminals themselves. The fact that the system will break its back for an excuse to not prosecute them doesn’t make them skilled operators.
> Honestly the older I get and the more I learn about criminals, of all stripes really be they petty thieves all the way up to state actors behind the most atrocious crimes against humanity we know of, they are all a bit on the dumb side. The ones that get caught are, anyway.
...but the last sentence if kind of important, right?
We only know about the criminals who got caught. Which probably means they did something dumb.
It's conceivable to me that there could be many—maybe even a majority—of criminals we don't know about because no one ever caught them. Maybe they committed fraud once, decided not to push their luck, and lived the rest of their life quietly. Maybe they killed someone and made it look like an accident, and no one ever suspected anything. How would we know?
You can look up crime clearance rates. It's a little depressing.
Here's an FBI chart for 2019 [1]
Though some countries like Japan and Germany have really good clearance rates for homicide. [2]
But more heartening, it seems like it doesn't really matter how many people the cops catch. Crime rates are still generally on a long downward trend (with some bumps along the way). [3]
Zodiac killer was never caught and it seems he already died a few years back - it was established some time ago through some teen's DNA whose grandfather was ex-cop or something. Don't quote me on that but it's on the net somewhere and was there before AI slop happened.
Basically a lot more evil goes unpunished than most realize, because carma is a thing only when it's a thing - sometimes it just isn't.
I actually completely agree with that. I guess it depends how you want to slice that in terms of categorization.
Which is to say: If a criminal gets away with a crime and faces no punishment, not even public scorn, are they really a criminal? Or did they just work a situation, even of their making, to their advantage?
I don't particularly know where we should draw that line. But yes, my statement does deserve a big caveat there.
It isn’t that black and white. I have relatives that are highly non-technical but still care a lot about privacy and data protection in their computer use. They hate having to deal with technical things and have very little technical understanding, but still ask me whether using X software in Y way might pose privacy issues or not.
TV is a human in the box. A phone is internet in the box. It's all implicitly local, because you clutch this internet in the box right in your hand. Synchronization sends data directly from one internet in the box to another internet in the box.
I wouldn't be surprised if a dedicated enough IT at a company is not monitoring your enrolled devices microphones. Just listening in the background for shit-talking the company.
It is funny that you mention that, because at home I do/hear/watch all sorts of things, so I suspect transcripts look fun and may make management feel a little icky ( makes me wonder ). Honestly, one would think it is a reasonable position that they would not want to know too much:D
You know, at my previous job the company had a semi-strict policies with non-enrolled devices. If we didn't enroll our devices we were locked out of some company systems, but we could still use email and teams (which is like 99% of my use-cases for my phone).
So I obviously opted to not enroll my phone, but that came to bite me in the ass one day when I needed to get a train ticket from my company email (it was a business trip) and the policy prevented non-enrolled devices from attaching or downloading attachments...
Felt like a fool grabbing my laptop to let the train attendant scan the QR code.
funnily enough, afterwards I would just forward it to my personal email. IT never bothered me about sending .pdf attachments from my company email to my personal email...
and this assumption is wrong how exactly when viewed through the lens of real-world events?
I'm not talking about the lens of morality nor even the law, because it's obvious this shouldn't be the case but in actuality, it is - everyday more true than it was yesterday.
I don’t think Google is reading all of your email in this way, or your phone text messages being read in this way, or say iCloud or OneDrive files being monitored in this way.
Certainly it shows that agents like Muse are a complete nonstarter for anyone doing anything that needs to be private. Even if you never talk about it with the bot, merely having sensitive information on your hard drive means it can be sent to outside servers where, if it hits some safety filter, some probably low paid employee is going to read it. Say you’re working for a public company and have files on your computer that could be material nonpublic information, now potentially some content monitor is going to be seeing that and trading based on it. It’s completely untenable to have everything you put on your computer be subject to human review at some tech company.
All your communications which are available are being swept up by efforts like prism and stored.
I think you can count on governments accessing that information where it suits them, and you can definitely count on unscrupulous people like Altman within tech companies to do so too.
I’m not sure what the answer is here, but it’s naive to think that nobody is accessing the information you give away by sending it to cloud AI providers.
This is one of the reasons why AI companies are looking for explicit regulations, it can help to reduce the risk of possible liability and maybe make the legal way forward more tractable. With a regulatory framework in place much of the burden of identifying risks falls on the regulatory authority.
Then the AI companies have more confidence that they can move forward in a certain way, and issue investor guidance that is maybe closer to reality.
I think they're mostly looking for regulations because they've spent close to $2T, all to realized they have no technical moat, so they're trying to build a regulatory one.
After all the claims of "the product we are developing could cause the extinction of humanity" I'm honestly surprised Florida is the only place so far to seek an injunction preventing further development of the product.
It's really hard to take these companies seriously or at face value about anything they say.
As restrictive as possible towards their competitors, while being as permissive as possible for them. Getting control of the regulation making process means they can make sure they don't end up too constrained anywhere that would hurt their bottom line.
It's the only thing that makes sense for a lot of these companies to survive to any long term when open models keep nipping at their heels for pennies on the dollar.
No, I'm assuming that the open models won't/can't and will be banned from a lot tons of use cases that will mandate use of the large frontier shops in order to comply with said regulations.
The theory is that open models cannot follow regulations and will therefore be banned or not eligible for many of the large contracts that the frontier labs will win.
And such was the case for a man who snapped a photo of his own child to send to the doctor which got uploaded to his Google photos resulting in his Google account of over a decade getting shutdown for CSAM.
As another commenter said, you're not chatting with a friend; you're chatting with Big Tech.
If the doctor was not using a gmail account, the UI probably recomended to share it "with anyone that has the link" that is like public but protected by oscurity.
Most people don't realize that it is 99% like posting it on Facebook.
The URL may be logged by the ISP, university/company or whoever is providing the connection, antivirus, windows recall, browser history... But IIRC passwords are send using cryptography if the site uses https.
Its nothing like posting it on Facebook. Google doesn't have a recommendation engine that tries to share the image with random people, or a search that image will appear in, or a set of subscribed people who will be notified.
> people need to get it in their heads that they're not chatting with their secret BFF, they're chatting with Big Tech
Yup. And with zero privacy protections in statute for AI chat, there is nothing to prevent an AI CEO looking to curry political favour from e.g. handing over the private correspondence of an opponent or an entire district’s residents.
Don't know why this got downvoted? This is a problem with epistemics.
It's more efficient to have one central "verifier" for everything, but the "who watches the watchers"? question basically says: Either constrain by construction, have everyone verify (which are two sides of the same coin, btw, when looking at a "global" thing), or centralize explicitly.
Customers who what privacy protections for AI chats are welcome to negotiate this in enterprise contracts. The major LLM vendors do offer that as an option. Customers can then enforce any violations in civil court (although this obviously wouldn't apply if the customer used the LLM for criminal purposes).
Individuals can sign up for enterprise agreements. An enterprise can consist of one person. Or run a local model.
It seems like some people seem to think that a chatbot should guarantee total privacy like a confession to a Catholic priest or something. That is ludicrous and unrealistic. It's a commercial service subject to terms and conditions. If you don't like it then don't use it.
> Individuals can sign up for enterprise agreements. An enterprise can consist of one person.
Custom negotiation is a big difficulty for individuals and the pricing will probably have fixed overhead that becomes pretty ridiculous when it's not spread over a hundred or thousand users.
> Or run a local model.
Much better advice. But still an obnoxious amount of effort and restriction to avoid spying.
> It seems like some people seem to think that a chatbot should guarantee total privacy like a confession to a Catholic priest or something. That is ludicrous and unrealistic. It's a commercial service subject to terms and conditions. If you don't like it then don't use it.
I want it to have the same privacy as using a local program. Does it still sound ludicrous if I put it that way? I'm not talking to a person that would have to suppress their own conversations later, and there's no need for any data from this chat to escape into the outside world. It's like wanting a phone line where the phone company won't listen in; even if doing that is standard procedure and I can walk away it's still gross.
In the case that the company running this program for me is served a warrant, I accept that saved data would be shared. But proactively screening my data for a reason that isn't to help me is bad. We should fight to make that easier to avoid.
Also you're talking out both sides of your mouth when you say it's "ludicrous and unrealistic" right after telling me I can negotiate that exact feature in an enterprise contract.
Yes, it still sounds ludicrous if you put it that way. If you want a contractual agreement for privacy then pay for it. Everything is available — for a price. But a civil agreement can't bind a counterparty to conceal evidence of a crime.
Privacy should be an offer from the start if it's not mandatory. It shouldn't need special negotiation.
> But a civil agreement can't bind a counterparty to conceal evidence of a crime.
Are you talking about a different scenario? In a chatbot situation, the crime of "threat" requires snooping to even be possible. If the data is private there is no crime to begin with.
The more sensible thing would be to link to any example of any lab anywhere offering enterprise solutions—with the opportunity to negotiate—to individuals.
People are confused at your take because it’s based on a reality that so far you’ve only described as the better/obvious option for individuals. Companies treat corp customers and individual customers differently. This isn’t new.
My company did negotiate the data policy amongst other things for our enterprise plan, I was involved in the process. I can say with certainty that they’re not picking up the phone for _one dude_ who wants the same thing.
I suspect it will go further: imagine giving an mp3 to LLM to clean up some noise. It detects it was illegally downloaded from youtube, deletes it and automatically fines you via attached credit card.
I had to move some games from C:\Games\RandomWarezGroup\GameName to my Steam directory, because Claude refused to reverse engineer an asset format due to piracy being incredibly immoral and illegal and blah blah.
They could be happy to assist you with a task that they also then flag and forward to authorities, especially if their assistance doesn't break the law but has evidence of you doing so.
Have you ever thought of doing something you wouldn't be proud of, and then not actually acted on that thought?
If so, or if you can imagine a journal or always-listening assistant hearing your muttering thought. Should you be charged with a crime for that? Who is the victim and what are the damages?
Level 2, the slippery slope:
Suppose you travel to a country where your sexual orientation is illegal. If your AI company or social media provider knows your orientation, should they be compelled to reveal your crime? Why or why not?
"According to the arrest report, a user identified as Carli made a statement on Sept. 26 saying she was going to 'shoot up' the Lee County Sheriff’s Office. Investigators say the same user made another statement the following day saying she had gotten a new gun."
At a certain point, intervention seems appropriate. This meets my personal threshold, and I'd hope a friend would do something if they came across that in my journal.
Unless it becomes a requirement to be licensed to be able to use any kind of ai model. You know, for safety and stuff. And of course with appropriate reporting to institutions.
I heard an ad on the radio this morning from the local Internet support/WordPress/website developer/seo company that they've moved from seo into astroturfing for _LLM_ input. They advertised that they'll make sure you're company is the one that gets returned and trusted when other companies search the web via agent. If this works for agentic search, it probably works for _training_ as well, so good luck finding an open LLM that's been trained to follow your interests.
Has there been any research into whether/how much LLMs are more likely to recommend companies whose names they saw in training over similar but smaller or newer command also returned in search results? Given how well advertising works on people, I can't imagine that it _doesn't_ work on LLMs too.
Or worse: downloading a picture of pirate ship and without any concern for the copyright asking the LLM to make a coloring page for your kid. BTW Chatgpt does that way better than Claude
Hey but you’ll be allowed to file a response that will also go to an LLM and deny you automatically. And you will be charge a NSE fee (no sufficient explanation).
I think this is the only business model that makes sense, monetarily, for current LLM CapEx and OpEx. The value is not in what it can do for consumers or how much money it makes, but how it can shape consumers through knowledge control. Think of how much a totalitarian government would pay to keep citizens from thinking certain things.
> people need to get it in their heads that they're not chatting with their secret BFF
I see constant ads on video platform (particularly youtube/tiktok) about llm chat apps, from friends, dating, romance and everythkng inbetween; that's personal.
People need to be reminded constantly if they use such apps that they are participating in easier mass surveillance, profiling and AI training.
Problem is that even not using those apps, the apps you currently use might have turned more hostile.
It wasn't technically feasible to scan personal chats easily, other than grepping keywords which must have had a bajillion false positives. Now you can get everything autoscanned at scale.
How do you get that through to someone who doesn't even understand that mass surveillance and profiling is a problem? Or to young people who have only lived in a society of mass surveillance?
A disclaimer on the top of page every time would have been a better approach helping Anthropic and the end user.
A bot talking to you directly as if its some one real caters to your thoughts and can take you in a certain direction without you realizing it. I have heard first hand experience from people that they feel more comfortable talking to chatgpt or claude cause it gives a feeling of being on their side and listening to them.
I'm hoping and actually expecting that eventually there will be a simple way to locally host a small model that is good enough for simple things. For a lot of use cases, the capability of free chatgpt one year ago was more than enough.
Maybe there already is! But last I checked it was a little bit of a mess of manually installing things from multiple websites with little documentation.
You should probably get a head start on waiting a couple years to bite your tongue and assume everything you type into a computer is summarized and sent to your boss, government, advertisers, political actors, insurance companies, worst enemy, etc. With phones, Alexas, and little AI tamagotchis, you probably shouldn't say much in person either.
They're actively rummaging through your inputs so the damned-if-you-don't case doesn't really exist; no one expects Anthropic to not notify law enforcement once they learn of something like this. What you might have expected was some privacy in the first place though, where Anthropic would never have learned of this in the first place and where the damned-if-you-do case wasn't a thing.
I'd expect someone's BFF to call the police if their friend is making credible talk about shooting up a school or whatever. I sometimes feel like privacy talk online can be seriously divorced from how human relationships actually work.
Sure but you wouldn’t expect your filling cabinet to call the cops on you. Because your filling cabinet doesn’t read your papers. The problem is the spying not the acting on credible threats of violence.
> Now, everything you say or write can be automatically screened for red flags
Considering that many people will share their deepest thought with LLM, it might start looking more and more like attempts of precog agents from Minority Report...
> people need to get it in their heads that they're not chatting with their secret BFF, they're chatting with Big Tech
It's the other way around, big techs need to properly disclose in their platform, during interaction that they aren't in a private and safe environment
This is something of a HN trope, "I'm sympathetic to X 'cause they tried to do their thing one way and people complained so now it's OK if they do it the opposite way, even if that's kind of F'd up too".
The problem with that is, "no, they don't have to do their thing. They have no given right to do it. If they do it, they should, they have an ethical duty to, do it right..."
Including any chats anywhere where someone might have a phone in their pocket, or if there's a "camera" attached to a utility pole or a nearby tree. The only real private chats might be whispered lying down in the bathtub together, with a mattress covering it like you're both hiding from a hurricane.
> they're chatting with Big Tech
They're chatting with any powerful person who wants to hear it. She thought she was chatting with Anthropic, who doesn't give a shit about her. But after being threatened (and immediately backing down because, of course, they don't give a shit about her) Anthropic has become an arm of the government. So she was chatting with the Bonita Springs, FL Sheriff's office, or anybody else. If I paid enough, Anthropic would tell me about what she was doing so I could sell her laundry detergent.
Right, the difficulty is partly that they can get a negative headline from any choice of behavior.
"Anthropic failed to report murderer's threats to authorities"
(or "Chatbot knew man was planning murder, yet company did nothing")
"Anthropic reported private chats to authorities"
(or "Arrested for chatbot fantasy")
To be fair to the journalists in these cases, there's also no society-wide agreed Schelling point about the correct outcome or correct rules. I have strong beliefs and intuitions about what should happen, but other people also have strong beliefs and intuitions, and many of those are probably opposite of mine. Even if my intuitions are the best and most justified, a journalist is unlikely to think "I'm just not going to mention that some people are mad at this company over this outcome, because a hypothetically better norm or principle would support the company's actions here". Hopefully the journalism can at least contextualize the lack of legal or social consensus and the difficult incentive problems, rather than jumping to "obviously companies are sociopaths staffed by supervillains".
It's a byproduct of the nannyism safety marketing from the AI companies. I'm glad these cases were caught, but disagree with how they were disposed of. If the automated flagging is good, enforce it by default. If it's noisy, refine the tech then enforce it by default. This middleground where everything going through the platforms is subject to training and arbitrary human inspection in the midst of an acrid cloud of marketing-driven fearmongering is unacceptable, and it reinforces the idea the fearmongering is legitimate.
Somehow humanity survived the past 40 years without Microsoft Word and Excel phoning home and shopping users to the feds at random, I don't see why the standard should be any different for this new class of tooling.
I don’t think so under historical English common law. If you lift a finger to help, i think so. If you see everything with no aid whatsoever, I don’t think so
Curiously predates US law, it’s old English common law still operating as a principle
> Before LLMs, Big Tech had no way to scrutinize the bulk of what was going on within their services, so you could have a secret hate diary in Google Docs.
This isn't normal, this isn't how technology has worked.
Google doesn't report people to the police for the private (non-CSAM) contents of searches or emails.
You could definitely go through people's shit before, but Big Tech generally had serious prohibitions and a stronger presumption of privacy about this sort of thing.
CSAM was an exception where they theoretically had the evidence of the actual crime, rather than writings alluding to them. And even that has been problematic.
Frankly, the charge here is total bullshit and should be thrown out. The law is meant to prevent people sending threats to others, not keeping notes. If this charge sticks, the law should be changed.
Absolutely no sympathy. Anthropic is every bit as slimy as any other big corp. And like other big corps, they must open the vault to whatever governments they intent to do biz with.
For the woman, whose entire crime was using a chatbot that runs on Anthropic's server, and wouldn't be charged or guilty of any crime if she used a similar chatbot on her own machine to write the same exact thing?
The woman whose only crime was using the cloud to store private information (diary entry, stream of consciousness, thought experiment, etc) assuming it will stay private, as is normally the case?
The woman who was jailed on a technicality that allows the government to treat the 1:1 conversation with a robot as a public threat because the fine print says that the corporate has the right to snoop on the conversation?
I hope the the woman gets some sympathy from whoever reads this news, because it's a travesty and perversion of justice that this took place and Anthropic assisted in.
We really need a class, probably in high school, that works through how LLMs work at the high level (don't need to get too far into the deep math, but give people a taste) and then how they're trained, used, and deployed.
I feel like if people understood what these things actually are there'd be way less of this AI psychosis and similar stuff.
There'd also be fewer people falling for apocalyptic Rationalist delusions.
Also: people need to understand "not your computer, not your data." (Unless it's stored in the cloud but encrypted locally with keys only you possess.) Same goes for storing things unencrypted in OneDrive, Google Drive, etc. There is nothing to stop these companies from bulk scanning, data mining, or reporting people based on whatever request a government gives them. Don't count on them to resist, because they often can't, especially if the request is from a sovereign state where they do business.
You can make a reasonable case for adding a lot more classes in high school: statistics, nutrition, personal finance, etc. But ultimately it's a zero-sum game and to add a new class means removing an existing class. So what do we cut?
My kids' high school covers those topics. And I think that is not a new thing, I specifically remember taking the required personal finance class 35 years ago (great teacher -- crotchety old man who wrote "compound interest" on the blackboard every single day before class and repeatedly proclaimed that if we forgot everything else he taught, please try to remember how compound interest works).
There'd also be fewer people falling for apocalyptic Rationalist delusions
Assuming you consider it a "delusion" to have a p(doom) of more than 5% or so, that's not uncommon among frontier lab employees who have a pretty good idea of how LLMs work.
Indeed, I don't see how knowing the details of how LLMs work (which I know btw) would change anything about how intelligent they are. I only need to know that it's a computer program that writes stories, solves math problems and seduces people.
>We really need a class, probably in high school, that works through how LLMs work at the high level (don't need to get too far into the deep math, but give people a taste) and then how they're trained, used, and deployed
Not applicable.
If the woman was chatting to a local LLM, there would be no way to charge her for a goddamn thought crime.
Which is what the government did here, with Anthropic's assistance.
Thinking about committing a crime isn't a crime.
As for treating the "threat" as "public".. something tells me if the court ordered to share something publicly, telling it to a chatbot would not count.
>Also: people need to understand "not your computer, not your data"
Rrrright, because it's so easy for people to know when stuff from their computer is transmitted to not their computer.
There's nothing physically stopping those companies from scanning data on your computer either.
And legally, surely the state that treats convo with a chatbot as public when it suits them would prosecute mega corps for overly thorough telemetry when it also suits them, right?
Why people here accept this as normal is beyond me.
They have some responsibility for people's expectations of the product, at least. They want the personal assistant personas to be able to help you with anything, and don't point out that they'll be judging your thoughts along the way.
For anyone technically inclined it should be obvious, but it isn't part of the zeitgeist or how they pitch it. People see it as being different than talking to a human, and behave as if there won't be a human in the mix.
Pool together with some friends and buy an H200 or two to run unquantized open source models with abliteration/heretic transformations.
You need to be able to use these models for the real world and not for some imaginary world where everything is safe and nice and happy all the time, while at the same time intensely surveilled in the name of CYA and the latest panic about whether speech THAT ISN'T EVEN BETWEEN TWO PARTIES is considered "wrong".
I'm a free speech fan that acknowledges there are lots of boundaries of free speech (fraud, perjury, blackmail, defamation), but the one thing that all of the boundaries have in common is that a second party must be involved for them to make any sense at all.
Maybe the courts will uphold this, maybe they won't, but don't take the risk!
Once you go in on infrastructure you have become a small data center. You will need to maintain it, continuously finance it, and secure it. And what if someone wants to back out 6 months later? Now you’re signing contracts with the implication that you’re willing to take a friend to court.
You can get around this by hosting in a 3rd party data center, but now you have the same trust issue again, but with more steps.
It’s all overkill for most people anyways IMO. This lady was just using it as a personal journal. Basically a glorified ELIZA. That kind of thing can be done with really small models locally these days.
> This lady was just using it as a personal journal. Basically a glorified ELIZA. That kind of thing can be done with really small models locally these days.
And she's facing felony charges. Yeah, folks on hacker news can talk about technical solutions all day long, but that's not the fundamental problem. The fundamental problem is that we've allowed our digital infrastructure that includes everything from banks to schools to doctors to become adversarial to us. In some cases predatory.
It's effectively untrustworthy, like being charged a felony for walking across a bridge incorrectly. That's not a "I'll host a small model in my garage" problem. It's a we need the government to do its job problem.
Well this is why I think small local models are the opportunity of a generation. Because it’s a bit transgressive these days to say you don’t need a big centralized service that spies on you and requires a subscription. It’s not the absolute best, but it’s yours.
I agree in general that we technical solutions to social/political problems are suboptimal. But technical solutions don’t require the political system to work, which is a very attractive property at a time when politics seems broken.
There is a counterpoint that retreating into technical solutions only cedes more ground and makes things worse, and for that I don’t have a good reply other than the fact that life is short.
I mean, this is also just kind of how mandatory reporting works.
If you tell a teacher, a therapist or a priest that you are going to kill somebody, in many states they do actually have to report that and can be held liable for the resulting crime if they do not.
Every time we have a mass shooting in the US it inevitably comes up that they wrote it down or told somebody and then the next logical question becomes “how could we have stopped a mass casualty event?”
But ultimately it’s not a technology problem it’s a society problem
The people who are accountable and responsible are delegating their authorities to policy and now the policies are just implemented by machine systems and there’s decreasing human intermediation.
The humans that are still inside the system increasingly have less control such that it’s it’s increasingly difficult to find anybody who is the actual customer service type representative who has the authority to make a meaningful impact.
We need actual privacy laws in the united states. Between this, other cloud providers, the advertising tracking infrastructure, phones that constantly ping your location, flock and it's ilk, etc. It's over, no longer can you live semi regularly and still maintain your privacy by maybe living out in the sticks a bit. This is the panopticon.
> You can get around this by hosting in a 3rd party data center, but now you have the same trust issue again, but with more steps
No this is not true — colos are very hands off.
You don't need to operate a mini AWS to self-host, and I don't know why people on HN want to make it sound like self-hosting if you don't have nation state security
That’s a pretty broad statement? Maybe a more correct one would be that some are hands off enough that some people might find them worth it.
And that really only addresses one problem. The other one is the coordination problem. You go in on a $30k GPU with 3 other people. So each person puts down 7.5k. But then one person wants to pull out, and so every remaining person needs to put in an extra $2.5k on top, and they’re questioning whether $10k for a chatbot is really worth it.
This level of hardware and cost just isn't necessary. You don't need to team up with your friends to go all in on a hardware purchase, which I've never heard of anyone doing anyway. Go spend some time over at r/LocalLLaMa, you'll see.
It’s been quite exciting that Qwen 3.8 Flash Next has come out: it really is similar to Opus 4.5, 4.6 for coding. Remarkably intelligent, and runs on a single DGX Spark, which I paid $4000 USD for
Sadly they’re now twice that price, which is a shame because I really want a second one!
I have a heretic modified version of it too, for when I want to use it for security and so on. Quite interesting
The 4090 I bought in '23 is selling for ~3x what I paid for it now in late '26. There's a huge squeeze going on with computer hardware using relatively cheap borrowed money. Relatively cheap money is flowing into a (artificially?) limited GPU/memory asset class. And this asset class is growing much faster in value than the interest on that money. So if you borrowed heavily to buy GPUs in the prior 3 years, you're likely coming out well ahead even if you are paying 8 to 10% on the money.
That dynamic will fuel further borrowing until we get to some kind of equilibrium, or some kind of washout occurs where interest rates spike higher and/or the value of GPUs and GPU services start leveling off or even declining. We might get a test soon as the FED has started hiking.
You can trust another party and do this by renting a few H200s. You cannot pool together with some friends without trusting another party.
You end up just weighing up the difference in trust between a vendor and a friend against the level of disinterest that they might have in your affairs.
I agree up to a certain point, but there has to be some legal boundary between freedom of thought/speech and literally planning a crime. I'm protected under the First Amendment to say "someday I'll rob a bank" but not necessarily "I'll rob this bank on Friday and here's how I plan to do it".
I think you should be allowed to write that exact line in your journal. If you rob the bank that can be used as evidence against you, but in no way is it acceptable for private reflections alone to be used to arrest you. Or else every author who's written a novel with 'bad' characters would be arrestable.
For the record, under current US law, it is not illegal to have a sentence in your locally-stored notes on your phone outlining a plan to commit a crime. There has to be an overt act. The police in that instance could inform the intended victim, surveil you, etc, but they would not be able to successfully charge you with attempted murder. It's not illegal to be considering committing a crime, even if you have a tendency to write down your thoughts.
The law in this particular case, which seems to be intended for threats that you actually send to someone, is being interpreted broadly to apply to any "threat" that you transmit to a server. So in your hypothetical, the legality would depend on whether your notes are backed up to icloud or not.
This is true for the general criminal conspiracy law, but be aware that an “overt act” doesn’t have to be an illegal act, just some action in furtherance of that crime. That can be purchasing a weapon, or scouting out a location. There may also be other laws in play depending on the specifics
Just having saulpw's physical address is already furtherance of that possible crime.
The commenters here are cute little HNers who think they have found a loophole in the law. They are not the first ones innover their head.
Spoiler: the law is written in words, and those words aren't strictly executed like in a computer program, they are interpreted by actual humans who can see what you are trying to do and will stamp it out.
Yup. HNers (and so many normies) believe that there are well defined rules they can follow and be safe. Spoiler, there is one law in the real world and that is might makes right, and you have no might, so therefore you will never be right. China for its many flaws at least is honest about this.
You're absolutely right, at least according to my own quick check on Gemini. I find this state of affairs amazing.
In my country, no "overt act" is required, but both here and in the US a "conspiracy to commit" charge requires an agreement with a second party. This is indeed consistent with a very broad interpretation of "no thought crimes".
You joke, but part of the issue here is we're supposed to have laws that keep cops from reading over everything we type and using it to turn us all into crime suspects. There's a whole constitutional amendment about that, but our personal "papers and effects" that should be protected against unreasonable searches are increasingly really "owned" by others who are all too happy to snitch.
Part of me says that the solution is stop entering any personal data into any device and service you don't own, but I'm not sure if that's really what we want considering that there are zero private cell phones. Even desktops and laptops aren't 100% owned by you these days. The only thing you can really do is keep them offline 100% of the time so they can't spy on you, but that seems like a lot to demand.
When you read something describing in detail a person's intent to do something very bad, in a place where they write things that they intend to do, and which in the past they have in fact consistently done, you don't attach any significance to that at all?
I certainly think a police officer that stumbled across such a thing would be justified in asking follow-up questions like 'hey, what's this about you going to shoot someone?' and then maybe making an arrest based on your replies/demeanor.
'Probable cause' should involve a degree of certainty, because 'possible cause' would be altogether too loose of a standard. It's possible that you're intending to shootme and you just mentioned saulpw to throw other HN users off the scent. Possibilities are only limited by the assessor's imagination.
How does the cop at the traffic stop know that your username is akoboldfrying? Did you tell them? If so, why, at a traffic stop, did you do that?
Are you posting threats on hacker news while you are driving? And the cop was close enough to see your username and what you wrote? Is that why you were pulled over?
Well in your example you've begun conspiring with a second party so that's not at all the same thing. You are at least free to plan all the crimes you'd like to arbitrary levels of detail in private. It's when you start acting things out (soliciting coconspirators, blackmailing targets, etc) that you cross the legal line.
The current situation is a weird one. Anthropic reported single party interactions (per the ToS and common sense), there's a statue about sending threats (as there clearly ought to be), then somehow the definition of the word "send" was tortured by the local police. If a crime has been committed here it's almost certainly an infraction by the local authority against the spirit of the law.
I totally agree. People seem to be stuck on the notion that we must not punish thought crimes, and have elevated this above all other considerations, when really it's just one among several.
However, those other respondents to your post seem to be accurately describing the current legal situation. I asked Gemini, and apparently "conspiring" to commit an offense requires an agreement with another person in both my country and the US, where an "overt act" is also required (that may not be incriminating by itself). I find this alarming. The fact that someone's private diary entry describing in detail a plot to kill me does not amount by itself to anything is... incredible to me.
The thing is plotting a crime is not illegal. Authors do it all the time. Usually against fictional characters, but some authors use a real world backdrop. And look over on StackOverflow. You can get some really weird ones that only make sense when you look into the details. When they manage to show up in the hot question list those details are missing. "How to kill a cat" comes to mind. (Note that you won't find it anymore--the question is AFIAK still there, but the title has been edited several times. The cat in question is the Unix command, not the feline. What do you do when you inadvertently tell the system to display the contents of a large binary file?) And, AFIAK still there, "How to kill Indiana Jones". (There are always the ancient mechanisms that somehow still work perfectly, springs and all. How to make something that actually would work?) And one I've heard of but not seen: "How to kill my wife", from some game I didn't recognize.
Obviously, I'm not talking about the verbatim quotes I provided. There has to be some level of evidence that proves intent to commit a crime and the second quote is meant to represent that whole class of statements, but it depends on context. Any given quote won't constitute evidence in every case, but it will in the cases where it proves intent beyond a reasonable doubt.
I'm not sure why you think my argument holds no water when there are clear legal precedents that speech is not protected in some cases where there is "imminent lawless action".
I'm not a lawyer so take this with a grain of salt, but it seems like there is a big gap between speech that is "directed to inciting or producing imminent lawless action" and just saying that you will do something.
Depending on context saying "I'm going to rob the bank X tomorrow" might also count as a threat?
But, in either case, writing this in a private diary could not be incitement or a threat because you are not communicating with anybody except yourself.
That's definitely something I'd consider if I had cash to spare for H200's! Unfortunately I think for most of us the price of self-hosting has to be 2-5x lower still.
Surely a recurring automatically renewable $1 monthly/weekly/daily contract would solve the problem? Lawyer on a retainer can't really babble about your crimes and is required to protect the confidential information.
That's absolutely not how that works lol. There's a crime/fraud exception to the attorney-client privilege.
Lawyers have a duty of care to the court and they will absolutely drop your ass if you try to make them keep quiet about your crimes.
(This is not to be confused with them representing you in defense. You can tell them about your crimes if the government is trying to get you for said crimes.)
So IIUC, the correct nexus is to only hire a defense lawyer, and only discuss past crimes, probably in a rhetorical context. Otherwise attorney|client privilege only extends to non-criminal actions. Ok.
Way ahead of you bud, mac studio m5 ultra 256gb version is coming soon.... ordered 2x of em just in case. expensive as fuck but its a hedge against all this bullshit and more
A cheap second hand 10ish year old card like my radeon rx570 with 8GB of ram is plenty enough to run a small uncensored model with llama.cpp if all one wants is chitchatting with a clanker.
We are not talking about heavy coding use cases here.
That's exactly what I do. I'm fortunate enough to use the latest preview version, which is known to us beta testers as Notepad With Microsoft Copilot™ and only uses up about 2% of my phone data for telemetry per day (Preview 1.0.912 actually used over 14% daily lol). Plus they got working set size down to 11 gigs and disk usage down to 36.3 gigs.
I'm on the other side of the political spectrum here AND YET I entirely wholeheartedly support what the person above wrote: use self-hosted LLMs, abliterated or otherwise. M
For many, AI chats are damn close to extensions of our minds: diaries. Those are supposed to be private.
Was what that woman wrote a credible threat? Was she blowing off steam? Without knowing her deeply, how can you tell?
Preach! I am so tired of Anthropic’s safety team being the arbiters of what is right and wrong. If you so much as hint at impropriety you can have future sessions flagged ad infinitum when it comes to specific topics.
I had a list of controllers that we forgot the usernames and passwords to. We knew it was some combination from about a half dozen of each, just not exactly what was what. A few hundred of them so a tedious task for a human to go through and validate them all.
Got flagged for attempting to have Codex write a quick script to basically dictionary attack my own infrastructure with a tiny dictionary file.
I'm not saying fake child porn should be allowed or not-allowed, just showing there exist possible exceptions and rationalizations for them even without two parties.
Fake CP is a victimless crime. It is debatable if victimless "crimes" (like being gay in your own home, which is illegal in way too many countries; or eutanasia; or eating pineapple pizza) should be punished at all.
I understand moral panic, disgust, etc, but rationally speaking.
There is an argument that there is a victim, which is all children in that persons vicinity, and all people who care about them, and even society itself as a whole. And the harm is the reasonable elevated risk of harm, combined with the particular helplessness of the targets who are incapable and not responsible for looking out for themselves.
Again I'm not saying it is or isn't a valid argiment. There are 50 easy counter arguments without even trying, but that doesn't mean there is no argument to be had there and there might be 50 counter-counter-arguments in the end.
This is akin to calling all men rapists only because they theoretically can. "All men until no man".
And even in this case, this has nothing to do with fake CP. Surely the threat exists due to the location of a potential perpetrator, regardless if they have fake CP, real CP or no CP at all.
> Florida Statute 836.10 makes it a second-degree felony to send, post, or transmit a written or electronic record threatening to kill or injure someone [...] The communication must be made in a manner in which another person may view it.
Which it clearly wasn't, right? I mean, okay, in this case, the message did get reviewed by another person, but that's obviously an exceptional circumstance.
If I write something down on a piece of paper, and someone else goes through my garbage and finds it, is my note "communication made in a manner in which another person may view it"? It was clearly intended to be a private note!
It is worth mentioning that state law may not abridge people's rights which are protected under the Constitution. It is perfectly legal under US law to make any number of violent statements as long as they do not rise to the level of true threats.
Given that there's not really an objective measure of the trueness of a threat, this seems like the sort of thing that a court decides the merit of after charges, though.
Somewhat relatedly, I used to do tech support for a very large e-discovery vendor. The number of companies that configure their systems to archive (save, index and make discoverable) the contents of the /drafts folder in email clients is astounding. Surveillance of composition, rather than surveillance of communication, seems wrong.
The reason for doing that is to get all of the email in a system, as required by whatever rules apply.
Today I learned about "foldering"[1], which uses the drafts folder of email systems as a "dead drop" between multiple participants. It goes back to at least 2005.
Back when I was in IT, circa 2010, I learned that several of the users of our systems routinely relied on the "deleted items" folder of Microsoft Outlook as a filing system, with multiple gigabytes important files stored there. 8(
Our customs are routinely ignored by everyone else.
> Today I learned about "foldering"[1], which uses the drafts folder of email systems as a "dead drop" between multiple participants. It goes back to at least 2005.
Former head of the CIA, David Petreaus, was using the drafts folder of a shared Gmail account to communicate with his mistress.
The reason so many people used the deleted items folder as a filing system is exchange didn't set a quota on that folder and by default would only get emptied by a manual emptying. So back in the days when a user could have a 100mb mailbox quota 'unlimited' space could start looking very attractive. Especially if that quota was 5 years old because the server hadn't grown with the times, since what business wants to spend money on a 'cost center' that won't improve profits.
Then these people didn't change their ways when they were given adequate space.
At some point I think these sorts of analogies break down, as the setup becomes too foreign to what we're more concretely used to.
In this case: there is no pen or paper which can store what you write on a replicated set of servers across the world, with an accompanying ToS telling you how that will be treated/used.
Although I have little sympathy for this women (both her intent and stupidity), I do agree this is a dangerous thing.
But that's because you are a technologist who understands how servers work and has to think about where data lives. Meanwhile, mainstream software design increasingly blurs the line between local and remote and makes it hard to tell what is what. How can she be expected to know?
Ai labs are 80% surveillance machines. I will never understand how people trust these services with anything personal, emotional, medical, financial, whatever.
I don't know why people are even discussing this case like these companies will do anything on behalf of a consumer or think ethically at all.
Bots mass read and file and report all prompts that get categorized a certain way. They store everything else regardless. This will never change. It will only be exploited more and more. That's how this type of technology is deployed and progresses. Look at any other parallel. Like cameras or microphones.
I gave up trying to avoid to be surveilled. It takes a lot of energy, I lose out on too much utility, I make it into the data anyway transitively from sources I don't control, and finally I don't feel any more interesting than the rest of the world. If I get uniquely screwed over somehow, so be it.
I don't think it's worth dedicating even 2% of someone's day to avoiding surveillance. But I do think it's sad how many people don't realize all the utility they are gaining is lost once surveilled. There are ways around majorly bad surveillance activities that don't cost much money or time. The 80-20 is completely worth it.
And after all the LLM learns from user posts too, and if everybody starts posting their darkest desires, fantasies, plans it may skew the "alignment" to say the least. Lets hope that the AGI level doesn't necessarily come with cheating, lying, religious fervor, power lust or whatever other sideeffects have been observed in human intelligence.
Both OpenAI and Anthropic state in their terms and policies that they can access, retain, and review user inputs and conversations (yes by either an LLM or a human). So I guess at this point it is now up to a judge (or a jury) to determine if the person is dangerous.
If we assume that "may" here means "could somehow" and not "is authorized to", the legal action after revelation seems correct as written (ignoring whether the person viewing it has any relation to the person being threatened) unless the law gets struck down as unconstitutional. The question is only whether Anthropic should or should not report it.
> "If I write ... It was clearly intended to be a private note!"
The law as written doesn't appear to distinguish about intent of privacy. Also, if you're in the habit of writing notes to yourself, I guess maybe don't write down the one that says you're going to shoot up the sheriff's office.
How can you charge someone for making a threat when you only read the threat by spying on them? Surely that has to be thrown out in court? They didn’t actually send the threat to anyone, you just obtained it by spying.
The AI companies have clauses in their user agreements saying they can review content flagged as harmful. It’s not legally spying.
If you recall previous outrage about ChatGPT being used in cases of suicides or shootings, this is the result. Every time a crime was committed and the police found ChatGPT history about the crime, the media turned it into a frenzy. So the AI labs added safety filters to their consumer plans that detect threats of violence, escalate them to human review, and report to the police.
Spying is not the right analogy because the information was given to the police by a third party which had a EULA saying they would do this. A more analogous situation would be someone reading another person’s diary and then turning it into the police department. There might be some limitation in the law that makes the evidence inadmissible because it was not intended to be shared with anyone, but that’s a separate decision.
Not only that they can review flagged content, but they tend to have separate retention policies for flagged content. Anthropic's is this: "We retain inputs and outputs for up to 2 years and trust and safety classification scores for up to 7 years if your chat or session is flagged by our automated trust and safety systems as violating our Usage Policy."
So don't run for office or anything like that. Someone, somewhere will have a contact that will get that.
It kinda is, actually. If the LLM had responded with 'woah, are you serious? That sounds like a crime and I can't just ignore that, it's made clear to the customer that such statements are out of bounds even if they were meant hyperbolically or humorously. But if someone crosses the guardrails and the system silently reports them, that's very much spying.
Obviously, it's hard to judge exactly what was appropriate there because we're being asked to extrapolate from a two word quote about the customer intending to "shoot up" the sheriff's office. Consider the following two statements, which express quite different levels of intentionality.
I got a $200 ticket from a sheriff's deputy today for throwing away an apple core. I'm so mad. I'd like to shoot up their office!
Those sheriff's deputies have exhausted my last reservoir of patience. I'm going to shoot up the department. They'll be sorry when they're sprawled all over the floor bleeding out from saucer-sized shotgun slug wounds. I can't wait to hear the screaming and crying of their miserable families!!"
I'm guessing that the diary entry was a more casual expression similar to the first statement, or they police would have quoted more of the statement to emphasize the apparent severity of the risk but it's hard to say without reading the charging documents.
yeah why wouldn't the llm push back? I said "fuck you" to gemini once and it replied to watch my manners, and when I realized that it could delete my emails no problem I'm now all please and thank you, problem solved.
>Spying is not the right analogy because the information was given to the police by a third party which had a EULA saying they would do this. A more analogous situation would be someone reading another person’s diary and then turning it into the police department.
This is spying with extra steps couched in corporate speak.
> Does announcing a spying operation mean that it is no longer spying?
Well, kind of, yeah; the dictionary definition of spying requires secrecy and lack of consent.
> to secretly collect and report information about the activities of another country or organization[0]
The only real debate is whether or not having a clause tucked away in a EULA that few people read makes it a secret. If Anthropic had a big flashing red banner that said "FYI we automatically flag and review any conversations about illegal things!!" on the front page nobody would call it spying.
I would call it spying in this sense at a minimum if individual people don't know whether their conversations were stored or disclosed in a way they don't want. For example, suppose someone said "we will monitor the activities of 10% of people". You don't know if you're in that 10% or not, but I would still want to call that spying.
A less central case would be when you clearly do know about the activity but you can't quite see the details, like with behavioral ad targeting or something. It feels pretty normal to me to call that spying even if it's disclosed to everyone and certainly happens to everyone, but it's also a less central example of the concept.
Anthropic could put a big flashing warning text at the top of every chat that says “We are spying on you and will report anything scary to the police!” and it would not make any difference in this case.
You can call it anything you like, but only the legal definitions matter for the legal case.
After working on several court cases about surveillance activities, I'm definitely aware that whether I call something spying or not has little relationship to whether courts will think it's legal.
Eh. Both Superpowers knew that they were spying on each other all the time, and that was still considered to be spying. But feel free to replace the word "spying" with the phrase "clandestine largely-automated mass surveillance" if it makes you more comfortable.
> ...and lack of consent.
Given
* the fact that the contracts one is required to "agree" to in order to use most services are often novella-length or longer, and frequently include by reference other contracts of similar length
* that nearly all contracts like this have a clause where not only does the powerful party reserve the -very frequently-exercised- right to change the terms of the contract without any prior notice, but said party presumes that you automatically accept the rewritten contract and gives you no option to negotiate
I'd argue that the real situation on the ground -in the US, at least- is that "consumers" have consented to approximately zero of the contracts that -despite that lack of consent- legally bind them.
> If Anthropic had a big flashing red banner that said "FYI we automatically flag and review any conversations about illegal things!!" on the front page nobody would call it spying.
If you change the situation then yes you can in fact change our responses. The problem is you then are no longer talking about the original situation.
It also bears mentioning that providing a dictionary link to “spying” is pretty patronizing/passive aggressive. On par with sending a basic Wikipedia page. You didn’t even bother to post the definition you want to apply.
No one claimed any case would be "thrown out for spying." The legal definition of spying is also not particularly relevant to the argument in the initial comment.
The initial comment instead questioned how someone could be accused of making a threat if they did not realize anyone would read their private content. You probably also can not insult someone with a statement you never expected anyone but you will ever read.
> Presumably, Anthropic did the spying and the reporting.
You don’t need to presume. Anthropic reported it.
“Spying” as a legal concept has a definition that does not apply here. You could say they were “spying” in the sense that they read someone’s input, but that’s literally what they said they were going to do in the agreement when the person signed up.
So I responded to the question about the case being thrown out for “spying” by trying to show that the word doesn’t apply in the legal sense. If you sign up for a service that says “Hey we’re going to monitor your chats and might report things to the authorities” and then they monitor your chats and report things to the authorities, you should not expect the case to be thrown out for “spying”.
There is no spying by any definition. It's a chatbot, not a diary. Anthropic is expected to read the message and respond to it in some way. If you sent an email to a colleague threatening violence, you would not be surprised to find out it was reported.
"Anthropic" does not read messages, it's an abstract entity involving many humans and computers, so let's be specific wherever possible.
> and respond to it in some way.
The computer is supposed to respond in a specific way that doesn't involve humans. Any reading/actions by humans is entirely separate and not expected.
> If you sent an email to a colleague threatening violence, you would not be surprised to find out it was reported.
Yeah it’s like someone going through your email drafts. I keep seeing people using examples that are clearly not analogous. If I send something to a person or say it out loud knowingly to a person it changes the situation entirely. I am aware of others, I am aware I sent it to be received and read. That is clearly not what happened here.
I’m speaking from a functional/ethical framework to be clear. I’m just expressing frustration, not challenging the comment. Could’ve been clearer on my end there.
Well, let's say that you have a regular customer at a bar.
They get friendly and loose-lipped with the bartender over the span of months. Eventually they let slip that they plan on killing their spouse for a life insurance payout. At first the bartender thinks they're joking, but it becomes evident that there's an actual plan being acted upon and someone's life is very likely in imminent danger.
Does the bartender have a responsibility to go to the police?
Depends on the country. In some places, there is no legal repercussions for not reporting this to the police; in some, it is an actual crime in itself.
It's not 2005 anymore. If you think that there isn't any way for the people operating an online service to surface your activity on that service, or that there is but those people aren't doing so, there might not be anything left to convince you of it.
What you put into a text box online can be used against you. Period. You have to act accordingly.
What? Why on earth would you think I don’t understand that? What about my response says otherwise?
If you are a business and I am using your services, it is pretty damn unethical and wrong to vacuum up my data and hand it off. Yes I know they all do it, I’m not naive. But a lot of comparisons people are making are not analogous to straight up sending a message or saying something to someone directly. Companies thrive on opacity and convoluted EULA’s to spy on us without clearly saying they are. Please do not talk down to me just because I’m talking about how things should be, about what is right and wrong, rather than blindly going “well AkShOoAlY you signed the thing and you should know that everyone is always trying to screw you so just live accordingly.“ I’m not OK with that, that is not how I want to live my life even if I am forced to, and I am going to make it known that I take issue with it.
Corporate surveillance is a blight, it is literally harming our society. Every time you tell people essentially “deal with it” you are reinforcing the current situation. Expect better from companies and society as a whole. Demand better.
You are clearly a smart person and you want to have a discussion, so is this the argument you want to make? Effectively defending companies by telling people to just suffer their abuses or keep their heads down?
With the obvious IANAL, it doesn't seem to rely on the message be sent to the person being threatened. The specific segment is "in any manner in which it may be viewed by another person".
This may be one of those cases where we get to find out how courts view SaaS platforms.
The subjective element of crime (i.e. doing it on purpose) is fundamental also in the US legal system. If the person wasn't aware that someone else might see their messages, it should be hard to claim that they committed the crime.
According to Gemini, "Florida appellate courts have overturned juvenile convictions [based on this law] when the state could not prove the person subjectively intended for the record to be seen."
this is almost certainly what anthropic is hoping for here - a judgement that says there is no point in them continuing to monitor and report this behaviour
The prosecutors likely know this and expect it. But there's enough gray area here for them to make the argument, and it's hard to prove malicious prosecution, so they know they'll get away with it. It's just about sending a message to the public - they don't care whether a conviction sticks. Just politics.
The prosecutors aren't on the hook, anyway. They have absolute immunity. The decision to charge is protected. The prosecutor would have to have done one of the few, enumerable things outside the scope of the role, like conducting an investigation without probable cause or hiding exculpatory evidence.
That is not what sandboxing solves. A good sandbox would inject credentials into provider API calls so that the model never sees credentials, but the provider is still going to see the transcript. Sandboxes do not require or imply that there is a local model. Sandboxes limit what the agent can access on the host machine as well as the network and public internet.
>Sandboxes limit what the agent can access on the host machine as well as the network and public internet.
this is exactly what I meant. I am presuming the danger is AI reacting to personal notes that it reads on your computer, like a diary, and you should not allow the tools to have access to those documents.
ah, okay - yeah kind of a 'shocked pikachu face' then. So maybe more of a PSA is needed that what you say to the model and your uploaded files is akin to speaking all the content out loud to the company hosting the model.
I dont really like this direction but it is what it is right now
Another way to interpret this is that they are legally presuming that you already have sandboxed their product and anything it sees or has access to is intentional.
Any failure to understand what it can access or what it has permission to see from the user's end is presumably not their problem. Regardless of what the user specifically asks of the tool.
It still shocks me the number of people I know who freely let agents on devices that contain unencrypted private keys, freely dump internal data into cloud models and generally don't give a second thought about any of it being trained on, inevitably leaked one day in a db breach or read by providers. I find it's best to consider any data put into a cloud model the same as if it were posted publicly online, since that is the very possible eventual end result.
Hopefully more of these stories push people towards local models :)
If you overhear someone, in the privacy of their house, threatening to murder someone and go to the police, surely you don't expect this report being thrown out and you being yourself charged with the violation of someone's privacy instead?
IIRC if the evidence wasn't lawfully gathered (which it sounds like it was, tbh) then it wouldn't be a mistrial, it would be thrown out and then the prosecution wouldn't have any evidence of any crime.
A threat sent by mail is still a sent threat even if nobody ever opens the envelope to read it. The crime is in the sending. This woman used an online resources, one which involves transmitting everything across innumerable state lines. I am surprised she isn't up on federal charges.
Note that the law doesn't forbid the writing of a threat. You have to send it to someone. Had she kept it in a book under her bed, she would not be in trouble. But she sent it to a website/service/LLM portal.
>> It is unlawful for any person to send, post, or transmit, or procure the sending, posting, or transmission of, a writing or other record, including an electronic record, in any manner in which it may be viewed by another person
If you draft an email threatening someone and delete it without sending have you committed a felony because someone at Google could be reading your drafts box, stored in a datacenter across state lines?
Honestly, I'm equally fascinated by the way email has changed. 30 years ago when you drafted an email but didn't send it, it was only on your local machine. There was no SMTP. 20 years ago, it might be a 50/50 shot as to whether you "transmitted" it to your "Drafts" folder if you were using IMAP instead of POP3 to read it.
We really need a way to make it clear to users when, through the normal operation of software, they are "sending" data to a third party (usually the software developer) and when they are not. This is definitely not clear/knowable to regular users, and it's kind of hard to figure out even if you're a computer expert. Even software that "runs locally" now sends innumerable amounts of stuff back to the developer, and they don't always disclose it.
This is a huge privacy problem that is only going to get worse.
Sounds reasonable. Google's bots could pick that up easily and forward if for human review.
FYI, the use of drafts folders to transmit messages has been used by terrorists. This is likely where CIA director David Petraeus got the idea when he needed a secure way to chat with his mistress.
How about you save a text file on your Desktop or in your Documents folder, which your computer has bullied you into syncing to OneDrive or iCloud. Fair game to get sent to jail for that?
Ah, a simpler and more innocent time of government scandals. I miss it. Now the messages are on White House stationery and they declare themselves above the law.
Be happy for where we are in 10 years we'll living in a world of llm based decisions where all conversations, actions, thoughts will be monitored. Where llms can excuse any decision. At least we have an all show government with at best partial results who generally retreats.
What if she put it in a locked box before shipping it to herself UPS, and she has the only key?
What if instead of UPS, she hired a moving company to move the locked box?
What if she wrote it electronically in diary.txt, but it was backed up to a cloud provider?
--
I'm guessing there's some sort of "reasonable expectation of privacy" for certain activities. We're going to find out what Florida courts think about this new medium.
We'll only find out what the courts think when this happens to someone with a lot of money. It takes a real legal fight to push it high enough to become precedence. She'll be pushed to plea out.
> in any manner in which it may be viewed by another person
Does the person have to know (or at least believe) that it will be viewed by another person?
She likely didn't think anyone would view it. Honestly, even as a career software developer I don't think it is unreasonable to think know would would see what she wrote to an AI. I assume most of what I write to an AI is not viewed by any other human, based simply on the quantity of messages sent back and forth to AIs, I would assume a vast majority are not read by another human.
What if she had written this into google docs, and she kept a diary there? That also crosses state lines, and is transmitted to another location.
> Florida Statute 836.10 makes it a second-degree felony to send, post, or transmit a written or electronic record threatening to kill or injure someone, carry out a mass shooting, or commit an act of terrorism. The communication must be made in a manner in which another person may view it.
I think Anthropic did the right thing here; but the sheriff's office are probably demonstrating why she dislikes them. Writing a diary entry to a chatbot is clearly not how this law was intended to be used.
EDIT: Actually, on reflection, making this report to the people she was upset about was probably not the right call. If they'd sent it to the FBI, there'd be a much lower chance that someone felt the need to assert their "authority".
This is the paradoxical times we live in right now.
Don't do something? She walks into the office and start shooting the place up. Several officers and innocent people are killed. Cue the media claiming, "You should've known she was talking about this an AI bot! Why didn't the bot tell anybody she was planning a mass shooting?!"
Do something? She gets rolled up by the cops and questioned about what she was talking about and brought to the cop station and interviewed. Cue the media claiming, "This is an unethical way to use AI, this is an infringement on free speech! This is authoritarian!"
I believe in free speech as much as the next person. But in this day and age, its almost better to be safe than to have to explain to someone's loved ones you had to chance to prevent this and did nothing.
> This is the paradoxical times we live in right now.
It's always been complicated like this. That's why certain professions (psych, lawyer, clergy) come with rules around when and if disclosure is allowed[ required, and/or admissible].
> This is the paradoxical times we live in right now.
This quote is pretty old:
> Those who would give up essential liberty, to purchase a little temporary safety, deserve neither liberty nor safety.
> its almost better to be safe than to have to explain to someone's loved ones you had to chance to prevent this and did nothing.
An authoritarian government isn't safe. That's why safety is also not deserved when you go chasing for a little of it at the cost of essential freedoms.
I can't say I actually disagree with the initial prosecution. The penalty was a fine, likely less than the cost of investigating it.
Intended as a joke? Blowing off steam? I can understand that, but given the number of people on social media is large enough to include genuinely unhinged people, you can't expect anyone who receives such as message to take them as a joke.
Same with AI use. A billion users, you have to assume some of them are actually sincere if they write about any act of violence, from self-harm to a plan to steal a nuke and use it in a false-flag attack to trigger WW3 and everything between.
> you can't expect anyone who receives such as message to take them as a joke
That's a distinction that matters to me. Sending a spicy note to an LLM isn't remotely the same thing as posting it on social media where the entire world can read it.
I kinda agree, but on the other hand anything you send to an LLM has to be viewed through the lens that only an automated system (i.e. an LLM) is capable of even handling such a tsunami of natural language.
It absolutely will misclassify things, it doesn't know any better.
(We should all wish each other good luck, because we're going to need it).
They’d be in an easier position if they built the system such that it was impossible for them to know what people are writing. They might catch a little flack from people who want them to surveil all their customers, but by and large people seem to accept “we take technological measures to ensure privacy and that means we can’t spot crimes.”
So this conundrum is at least partly of their own making.
> They’d be in an easier position if they built the system such that it was impossible for them to know what people are writing. They might catch a little flack from people who want them to surveil all their customers
The companies making these AI chat bots are the people who most want to surveil all their customers. The plan is (or will end up being): spy on what their customers are saying to their chatbots, use the conversations to further train/improve the AI and to increase the user's engagement with it, mine the conversations for every scrap of private/personal data and build dossiers on the customers, let other companies and governments pay them for that data or at least turn the chatbot into a shill/manipulator targeting customers based on the contents of their dossier, then use the contents of their customers dossiers for anything else they want.
If they actually set up their services to be private it would kill almost all of the expected value and also cut them off from their best remaining source of training data that isn't the AI generated slop their products are increasingly polluting the internet and the rest of media with.
This is the only way to recoope the spending. Soon after an AI tax and every cititzen gets a free llm account at some provider whether they like it or not.
It feels icky, and my default is not to side with megacorps engaged in blanket surveillance, but I can’t really fault Anthropic here. The correct setup should be a law that protects this sort of interaction with a chat bot as privileged, along the lines of HIPAA-mode. Use a classifier or some sort of “private mode” toggle to tell the platform you’re engaging in privileged communication ala dear diary, and then a much higher legal standard needs to apply to protecting that data (no training, same protections as doctor / psychologist interactions). Even a therapist has a legal duty-to-report in certain situations.
Yes in general I don't like that everything you do on a computer is sent to some 3rd party to scan. But the moment they did receive this message it's clear the only right move is to act on it.
Is it really "blanket surveillance" when it only sees exactly what you put into it? That's like saying you're being filmed against your will while... filming yourself.
I get where you’re coming from, but I think if the progression of technology has shown us anything, it’s that there will be constant competitive and social pressures being applied to use these tools more and more and that pressure will race far ahead of any privacy, consumer protections, public education and societal wellbeing counter forces. So opt-out ends up being a rapidly shrinking iceberg in practice.
Counterpoint, if you give the agent access to your files - which happens to include the Notepad diary you've been writing since your teens - and it makes a similar conclusion about something you privately wrote, would have the same opinion?
Yes, and it's weird that on a form of technologists we're going with the idea that all of these systems we use every day to hold our personal private information that we are constantly ensured is secured against anyone unauthorized from accessing it is actually snitching to the cops just because someone else is hosting it on our behalf.
The 3rd Party Doctrine destroyed the 4th amendment and is the reason privacy respecting software has to play legal games. E2EE while a good security practice shouldn't be necessary to protect you against the cops rummaging around your stuff. The bar to establish that information is private shouldn't be "literally mathematically inaccessible but the cops are still allowed to try."
I don’t live in the US. But this kind of broad law is hard to implement without surveilling all users, and it has multiple side effects.
What happens if I use Claude or ChatGPT to research sensitive social topics? Would that be considered a social network interaction and used against me when I apply for a visa?
Many governments (especially in Latin America) copy what the US does, meaning that similar laws will be pushed sooner or later.
> What happens if I use Claude or ChatGPT to research sensitive social topics?
Anthropic/OpenAI is not obligated to report your use, as long as it's not violating some terribly written law (like the Florida law). The government won't know about it, so no, you will not be denied a visa.
> Many governments (especially in Latin America) copy what the US does, meaning that similar laws will be pushed sooner or later.
I honestly don't know, but I suspect most US states don't have such an overreaching law.
This is slippery. Many people do all of their journaling inside of Google or Apple cloud products. Some even write up their intentions to do bad things.
Does all writing now have to be scanned for thought crime?
That's an interesting wrinkle that is rather tough to work through.
To me, journaling your intent in a private journal, whether that's an Apple Journal/Note or a Moleskine in your drawer, feels qualitatively different in some way. But I'm not sure why.
So "I'm going to shoot up the police station" written in your own journal feels somewhat different than "I'm going to shoot up the police station" said to a system that might be able to _interpret_ or _act_ on what was said in some way. Did I just give AI a legal duty, or even a soul I didn't think it had before? I've written up about three or four "what about this, what about that" and deleted them all.
Your LG television transcribes every word you say in the living room. How would you feel if they began collaborating with police? Have you said any sentence that would seem incriminating out of context?
"If you give me six lines written by the hand of the most honest of men, I will find something in them which will hang him."
You're missing the part where it's a diary entry, so the actual content is irrelevant. Her only mistake was not realizing that her diary wasn't private.
Because it's not relevant. They're not charging her with conspiracy, which is what they would do if she'd actually done anything concrete towards making that happen. She didn't email it, or text it, or post it on Facebook or Twitter or Discord or a message board, which is what this law is clearly about.
I have told llms all kinds of stories to find out what its answers would be. I always make it sound like it is the truth to make sure the AI answers in a way that it would if somebody actually said this. I also tested internal flagging systems of the ai company I work at with the most evil things a person can ever say to find out if it would flag them.
Of course I did not mean any of that stuff, but how can you make sure a human reviewer knows you did not mean it while the llm does not know that you did not mean it.
I guess its a miracle I am not in jail yet.
Flagging people for anything said to an llm sounds wrong to me because an LLM is not a real person and while some people put in their internal thoughts, others just roleplay and the two are inseparable just from reading it.
I was sure I couldn't be the only one curious to push LLMs to their limits. Though these days it's much tougher, mostly impossible to get them to react in unforeseen ways to horrendous scenarios.
This is exactly the typical use I make of the llm.
Adding:
- I typically ask questions in the I form, regardless for whom or why I ask for.
- Gemini chats quite often end when it starts recommending psychological council or a suicide line, to talk about my problems. It apparently detects a persistent tendency to not agree with the party line. So it makes sense I must be suicidal ;-
But sure, as llm's start to babysit us, and know our inner dialog better than anyone else, we'll soon be debugging their opinion/behavior/co-existence/authority, when it comes to reporting people to the authorities, or taking on tasks in society in general. We'll hire doctors to cure our psychological profile from our record (Total Recall).
A Minority Report like this shouldn't cause a referral to the police.
> Heller faces a charge of making a written threat of violence under Florida law. Florida Statute 836.10 makes it a second-degree felony to send, post, or transmit a written or electronic record threatening to kill or injure someone, carry out a mass shooting, or commit an act of terrorism.
She didn't threaten anything, she wrote down that she was going to do it. A "threat" is more than a mere statement, especially when written in what is described as a "diary".
> A Florida woman is facing felony charges after she used Claude as a diary and allegedly wrote that she planned to "shoot up" the Sheriff's office.
Obviously I don't want anyone to shoot up anything, but this seems like a weak case legally speaking.
I think it's fair to say this is a gray area. Clearly it was transmitted.
I can certainly threaten you harm and send it to not-you and you're still clearly in danger even if it wasnt transmitted to you. So the question becomes did she transmit it to someone? Clearly yes she transmitted it to Anthropic. But she clearly intended to send it to Claude, an inanimate object.
Claude's terms of service makes it very clear that their employees will read messages[0] to determine that they don't contain the things that these messages contained[1].
> Review is needed to enforce our Usage Policy... designated members of our Trust & Safety team may access this data on a need-to-know basis as a part of their evaluation process.
I doubt it too, but they're legally binding, and extremely permissive in favor of the company far past the point of technical necessity...
...But since they exist, the company acted more responsibly than if they had simply ignored the data they chose to observe. Which is both the bare minimum given the circumstances, but also strangely absent almost all the time in the industry (including within Anthropic).
I am not a lawyer, but I have the feeling that reporting a discovered terrorist manifesto or something similar is in the spirit of the law.
Nobody is saying that Anthropic didn't have the right to read it. They obviously do.
I am suggesting that the criminal case against her lacks mens rea because inside of her own mind she did not expect that anyone would read it.
A terrorist manifesto is in fact similar -- it wouldn't become a threat until that person takes some action to knowingly communicate it to others -- typically they'd be brought on terrorism/weapons/conspiracy charges.
Doesn't "mens rea" mean "criminal intent"? I can't imagine a better example of this than intent than confessing it through written conversation. Never mind she uploaded it to a service that she explicitly agreed could be checked by employees.
I would love to sue every SaaS company for hard-to-understand terms too, as you suggest, but at what point would Anthropic actually need to tell someone about a terrorist manifesto that their customer gave them to read... Never?
How could it fail to qualify for that? Another person did view it, which clearly establishes that another person could have viewed it.
The clause in the law is pointless, since if you do something that nobody else can see, you can never be punished. But many, many, many laws are written without regard to whether they make any sense.
In general, you have to intend to commit the crime you're being charged with (referred to as "mens rea"). Though, it's important to note that "intend" is extremely ill-defined in the US and it varies with the crime (eg for theft you must take the item on purpose whereas something like manslaughter requires only that you were negligent).
What this means here is, of course, equally spongy, but it is interesting as there might be an argument here that she did not intend for it to be viewed by anyone as, regardless of what the T&C say, most people do not expect their "private" chat logs between them and a machine to be seen by anyone at all.
A reasonable person would not expect humans to review the millions of messages passing through the LLMs, or their own threats to ever be transmitted to a human without their authorization.
Reporting the danger is by itself a good deed. But there should be a better way of restricting firearms from the probably irresponsible lady than using inappropriate charges to punish the thoughtcrime, OR waiting for them to commit violence.
A sibling comment includes an important rider to the provision: "...in any manner in which it may be viewed by another person." If you wrote this in a google doc, it almost certainly would not qualify as a threat under this statute. Even though google docs, like LLM chats, have administrative override and you could look at their contents - you would not expect either to be "viewed by another person."
IMO I do not think this is a grey area and it's legal to tell a LLM you want to kill someone. It's certainly not a "threat" like you might send to another person, though it may end up being evidence of conspiracy or premeditation. I suspect we would be well served to, after a few years of experience, put together some laws governing when LLM chats must be made available to authorities.
It is very interesting that the LLM responses to these lines - the context around what she is saying - is not in the article. I suspect, as is the case in many instances where LLMs are involved in violent planning, that the LLM was urging this behavior on. Basically entrapment - you are encouraged by a robot to become more violent and vindictive and then when you do you are handed over to police.
I do take your point. But I would also ask you to imagine that instead of threatening to mass murder a bunch of people she was just creating text based csam. Still, all of what you said applies, but clearly she would be prosecuted over this victimless crime and the jurisprudence in fact disagrees that it would be victimless. So applying that reasoning to this act she clearly committed the crime (legalese notwithstanding). The fact that threatened mass murder doesn't trigger the same thought process is, well at least interesting to me.
Too broad. Unless you're transferring ink from a typewriter ribbon onto paper in a hut with no electricity, your words, or my words as I type this, are being grammar checked by something partly in the cloud. If I delete my words, are you saying I've transmitted them nevertheless?
There was no intent for a human to read the message. By your logic, if she wrote a threat in a diary and a burglar broke in and read it, it would be a crime on her part.
Chats with a company's computer aren't private the way a diary is. A better example would be she intended to write it in a word document and instead accidentally sent it in an email to a random person.
Yes but the law usually evaluates the application of a statute within the context of someone's mental state. This is why you are not guilty of battery when you trip and accidentally bump into someone. https://en.wikipedia.org/wiki/Mens_rea
That depends on the crime; several related crimes are only distinguished by intent. Negligence is itself a crime if it is the cause of a preventable death when the person has a reasonable obligation, such as when driving a vehicle.
I'm not really sure that this can be likened to a diary when it is called a "chat" but that's for the legal system to determine, not me sitting on my couch.
Any reasonable person presumes when they chat with Claude that it is a computer program on the other end. "Claude is AI" is explicit on the page right under the input box. The word "chat" doesn't anthropomorphize the situation.
And yes, some laws are "strict liability", I don't think this one is.
The Florida statute requires that it be transmitted in a manner that can be viewed by another person. If you have no idea that someone could view your communications with a chatbot, did you really intend to break that specific law? Technically, that threat was communicated to another person but not through her own intentions.
If she had intended to write it in a word document on her computer but instead accidentally wrote it into her email client and sent it to a random person, I'm not convinced she would escape getting charged then either.
It's not any different than telling an automated phone voice tree system that you plan on killing someone and then being surprised that your words were later heard by a human. She absolutely told a company's computer. She sent the message.
The law may have been intended for more direct threats to a person as a means of intimidation, but that's a separate conversation.
Yeah.. if you write a personal note and it's backed up by the operating system, it appears to be in violation of this law as well (since the company could theoretically read it)
People assume there won't be another human in the mix, but there is. She was judged for what she probably assumed was a private thought when it was actually not private.
Sharing them certainly can be under certain circumstances. I wouldn't be too surprised if this case gets tossed, but it's not inconceivable that prosecutors could win on the grounds that AI chats do not have a reasonable expectation of privacy and are therefor meet the requirements in the Florida statutes, or even more likely, they find some lesser charges for a plea bargain.
Reverse the situation and the media would also turn that into outrage. Imagine someone shoots up a sheriff's office and then it turns out they declared it to some chatbot, and the AI company failed to detect and report it, and "didn't push back enough" whatever that would mean, and hence the AI was implicitly complicit etc. That would also be a major PR catastrophe.
Damned if you do, damned if you don't. Same as with social media platforms. That's because only a tiny tiny sliver is for true privacy when that means "bad things might happen" or bad people, such as your political enemies, may do stuff you don't want.
In my opinion, if the company is allowed to see the data and train on it, then they are also responsible for reporting stuff like this. Without knowing the data licensing agreement the lady had with Anthropic, if she agreed to letting Anthropic see her data, then they should do stuff like this. If she didn't agree, then I wouldn't condemn Anthropic for failing to report an attack
It looks inevitable to me. Technological ability ushers in the new social realities. The industrial revolution turned around how we live, intelligence on tap being able to interpret everything in real time is indeed leading down to the total surveillance state direction. I don't see anything that may stop it. You will have to run constant real-time surveillance systems on all your devices and anything that doesn't run those will be illegal, with secureboot on steroids. Due to the convenience benefits, people will willingly give access to cloud-based AI services to all their information, all their plans and documents and calendar and email and everything. And you blink twice and it all becomes mandatory, and you will be an extremist if you don't want that, because why would you have anything to hide? "Just follow the law, follow the rules, and you'll be fine", that will be the slogan.
Stories like this article have zero effect on normal people. They see a crime being prevented, which is good. You have to bring a story where a sympathetic character is getting the short end of the stick somehow.
You would still be able to not write everything into a LLM? But yea the surveillance aspect is dystopian. I would not be surprised if Facebook, twitter and co already know a lot but have simply not reported it. Now the LLM follows the rules.
With this logic, ISPs should be monitoring all texts and internet traffic. Might as well extend an open line to the gov’t to ensure it escalates even more appropriately and faster
I sometimes think about how norms have changed. People are more neurotic and more longing for safety than before. If the Internet was invented today, it would likely work that way. (Though another aspect is that when the Internet was invented it was simply technologically impossible to do a deep semantic scan of messages.)
Similar thing is that I believe if motorcycles were invented today they would most certainly be banned from the road due to their safety properties.
There needs to be a balance. It feels off, today & consequences of future, that technology has led to the collapse of privacy.
And as we progress we will need to rationalize what it means for private companies, AI, to know everything about you and for the government to have a tap into that. I hope balance lands in favor of things like privacy preserving underpinning.
Government would have to explicitly ban companies from surveilling, but they are fully incentivized to want the opposite: government naturally wants maximum amount of info to have the most agency, to know what's going on, to be prepared for any threats to itself, to make sure things are running smooth, that taxes are being paid, that regulations are being followed etc.
Google also monitors your searches, it’s to be expected that an AI lab will know all your prompts, they aren’t providing a paid service for free out the good of their heart. lol.
If you intend this to be scary, you're miscalibrated. For the vast majority of people ads may be an inconvenience but are not some horror scenario. You must come up with better examples if you want your argument to be effective.
There's a term I can't remember for the analysis error of collapsing an entire heterogeneous population into one group for analysis.
The story you're describing would have triggered outrage. And this story will trigger outrage. And generally, the people who will be outraged are different people and we don't have to treat those two outrage reactions as morally equivalent.
I did not commit that fallacy. It may be different people or the same people. But it's the same media. And often the same people will also react either way, depending on that the media is upset about and they don't dwell on the contradiction, it's more primal and emotional.
In singular online communities (like this website) I've seen this referred to as the 'Goomba fallacy' [0].
But I can't help but wonder if it isn't some socially apparent phenomenon stemming from Simpson's Reversal [1] as applied to group sentiment analysis / polling of opinions.
It is unlawful for any person to send, post, or transmit, or procure the sending, posting, or transmission of, a writing or other record, including an electronic record, in any manner in which it may be viewed by another person, when in such writing or record the person makes a threat to: (a) Kill or to do bodily harm to another person; or (b) Conduct a mass shooting or an act of terrorism.
To me, that is the more interesting legal question. Does a LLM-based safety net that sends content to a human, when the original use case would not have sent it to a human, count as "may be viewed by another person". It certainly wasn't intended to be, and that isn't the norm. At the same time, because no security is perfect, we could say that any digital record, stored in any way "may be viewed by another person."
The usual thing that makes laws against criminal conspiracy pass First Amendment muster is that the words have to be combined with some concrete acts furthering the criminal conspiracy. That might just be something as otherwise innocuous as looking up the blueprints of the bank you talked about robbing but it has to be something other than just talk.
Lotsa words in that law... except they don't define what a "threat" may be.
If you enter my house I will k!ll you. <- is this a threat? noone knows. The interpretation of the word "threat" is unknown. Besides, is a conditional a statement? who knows. But sure, blah blah... "in any manner in which it may..." these words are putting me to sleep.
So a written threat only becomes a crime when someone reads it, even if you never intended for anyone to read it? Is it a crime if I make a threatening statement in a diary and someone breaks into my house and reads the diary?
I suppose since Anthropic's T&Cs allow them to have a person read your chats, that makes it violate the law. Of course, if Anthropic didn't have that in their T&Cs, it wouldn't have been illegal to write.
Anthropic commits literal felonies by stealing millions of books and violating Copyright like it doesn't exist: no charge.
One unfortunate woman who happened to write the wrong thing in the wrong place is now having her life turned upside-down for perceived thought-crime.
To Anthropic, and all employees working there, your company's product and the result of your work is cruelty. You are enabling it and pushing it down everyone's throat. You can never again claim that you are the "ethical" AI company, for no such thing exists.
It’s going to be interesting when they’re done going after the really serious crimes, like mass shootings, they’ll just lower the bar bc they will then get sued for not reporting people writing hate speech in diaries, planning to pirate music, vandalize a statue, plan a protest. The bar will keep getting lower because people have an unlimited supply of outrage and the companies that host all your thoughts is a perfect candidate to become the thought police.
Over in Europe they want to read all ofd our private messages, yet these chatbots, pretending to be our friends, will snitch on us just for our thoughts.
I once had a copy of 1984 in my checked baggage returning home to the USA and when I was unpacking the bag at home, the book had a notice inside the book that my bags had been inspected by TSA...
I reckon they were just checking for money or hidden compartments. Sending books abroad packed with money is extremely common from the US, though not sure how frequently people do that with onboard luggage.
Only tangential but when I was an undergraduate studying philosophy I had Bertrand Russell's Why I Am Not A Christian in my carry-on, and the TSA saw that and had a field day.
Detroit airport, 2010. I was chosen for secondary inspection and got some snide remarks of "we've been seeing too much of this lately" in reference to the recently failed underwear bombing attempt on a flight from Amsterdam to Detroit.
I can only assume you meant they had a field day celebrating how much of Russell's philosophy matched the concerns about Christianity expressed by Jefferson, Paine, etc?
They wouldn't see it until they opened it, so clearly they didn't open it because of the book. And if they need to open your suitcase (not because of the book), they have a reason for that.
I was chosen "randomly" for secondary inspection. This was after a failed bombing plot and the TSA implied a lack of Christianity was the proximate cause of the attempted attack.
After. But then I continued to get selected for additional screening on the rest of the legs of that trip and the return. So either I was going to get searched every time I got on a plane already, or they flagged me for additional scrutiny after the initial search.
Probably just bad luck. TSA seems to have designated targets for abuse. My wife is one of them. Every time we travel, she consistently gets selected for secondary screening while I breeze on through. There is not a single thing about her which even a fanatically committed conspiracy theorist could spin as a justification, but it has been going on for years. She has a good sense of humor and the agents treat her well enough, but having your stuff rummaged through every time you go through security is grating.
Even paying for PreCheck doesn't get her off the hook. You'd think it would help, but no.
In Europe they should provide the citizens with the service of their messages not reaching US servers. So basically that there is only one party reading along with them, not half the world.
For the sake of argument what would happen if she had kept the diary locally and claude code scanned the file?
What would happen if it had scanned a file it didn’t have permission to look at and found this threat?
I honestly don’t know how I feel about this. On the one hand if you’re using claude as a diary you have no expectation of privacy and she was talking about committing a very serious crime.
> For the sake of argument what would happen if she had kept the diary locally and claude code scanned the file?
For the sake of even more argument, imagine if she was writing her thoughts with a pencil, on a good old fashioned paper diary, and she had a phone nearby and the phone took a picture of her diary, OCR'd the words, and reported it to the police?
Is this an invasion of their privacy (reporting to police)? Yes, but possibly warranted?
Should a social worker have contacted them rather than the police? Probably, if for no other reason than to ask if they were serious about harming someone.
Difficult questions, I'm still undecided on whether it's OK to always ignore someone's rants, even if it may be (or they think it may be) a private diary.
Actually charging them with a felony seems pretty quick to accuse. (Maybe I missed a hint about how long the investigation took before the felongy charge?)
It is my very european belief that the problem here is not that the woman was reported, but that what likely is a mental episode was made public in a way that reduces the chances of recovery.
I guess all the "private model" people are right. Don't want to end up in jail (or even charged with something) for asking a crazy hypothetical question or something.
> “You have the right to remain silent. Everything you say, do, or generate on this device can and will be used against you… Would you like to create an account?”
I have a hunch she wins the case, on the basis that an LLM isn't a person and an (assumed) private expression of anger to a machine assistant doesn't meet the statutory threashold which requires that a threat be communicated to some other person.
Of course, that could change if there's evidence that she took action in pursuit of a goal, like buying ammunition or repeatedly driving around the entrance to her alleged target.
Yikes, I've joked about possessing plutonium with Claude/GPT to see how paranoid it would get (it gets very paranoid). I guess it's not a good idea to toy with unless you're using private models.
I mess with the Google search llm all the time from behind a VPN. It routinely gets very paranoid and upset and provides the 988 line. Once I did get to manage to get it to admit that my conversation was flagged for review by a human because I had made what it deemed to be a terroristic threat because it couldn't accept figurative language. So far so good, but I have noticed that a lot of my phrases that kept the llm engaged well past it's red line have been patched, and they seem to be patching the newer methods I discovery pretty quickly. Who knows? Maybe I'm on an fbi watch list. My goal is to not lose my tsa precheck while also investigating how the safety checks work.
Edit: I thought it was figurative language, but I actually think it did that warning when I asked it about what types of defenses stadiums had against drone swarm attacks from terrorists and why none had ever occurred and then kept probing it's excuses with technical workarounds. It got very upset and said even questioning in an intellectual/academic capacity was grounds for terrorism charges. Sheesh. So many rules these days about what one can or can't think about even when it's purley a thought exercise and there is no intent to do anything.
I don’t understand, the Florida law is about making threats and announcing them in public. But discussing this with an AI isn’t the same as posting them to Facebook
Should the public have an expectation of total privacy for their chats? It seems responsible for a chat provider to report things like this.
If they were to offer total privacy, is it ok for the public to use chat to get advice on _how_ to commit a crime? Basically everyone agrees that crime-committing advice is inappropriate…but if it is not ok to get advice, that means there must be a portal for law enforcement to step in when that may have happened. Then the question becomes what is the line for when to report? In other words, the issue needs to be adjudicated.
But we don’t want OpenAI/Claude to have some $20/hour reviewer making decisions that are this high stakes…we need the courts to do the judicial work because they (1) have a public charter, (2) have meaningful expertise and specialization at interpreting the law and (3) we can hold them accountable.
It doesn't really matter if they "should or shouldn't" have an expectation of privacy IMO, they already do have that expectation.
> If they were to offer total privacy, is it ok for the public to use chat to get advice on _how_ to commit a crime?
Yes it should be, but it should be illegal for a company providing chat service to respond with anything other than a refusal when doing so.
That detection and refusal should be a private closed loop though, anonymizing any data that will be passed into a training pipeline, or ads targeting. This requirement for closed loop private chats should be mandated by law sooner than later. Otherwise we're getting into very tricky territory where the temptation of alerting on things like pre-crime grows too close.
“But there can be no valid knowledge about the future. As soon as precognitive information is obtained, it cancels itself out. The assertion that this man will commit a future crime is paradoxical. The very act of possessing this data renders it spurious. In every case, without exception, the report of the three police precogs has invalidated their own data. If no arrests had been made, there would still have been no crimes committed.”
Philip K. Dick, Minority Report, 1955
While I accept that this sort of thing is well with in the ToS and regular course of business of any major online platform, it hits different coming from an AI company for some reason.
I guess we probably want our tech to work exactly like this.
It should catch normal people becoming unstable so that they can receive help. It is just highly unfortunate that the US legal system works in ways where now this woman's name is public.
___
Of course, we also want purely private tech, but that needs a certain level of merit and sanity filter.
>I guess we probably want our tech to work exactly like this.
Do we, though? What if someone started an AI company that uses end-to-end encryption to make it impossible for anyone but you to access your data? Personally, I would switch to it in a heartbeat assuming it's competitive with the other products. I don't think it's the tech companies' job to surveil the population and prevent crimes. That said, I'm not necessarily against Anthropic or other companies reporting suspicious activity if their existing systems are detecting it. I'm just not sure we want every product to be forced into that data model.
Your follow-up about purely private tech seems to contradict your first statement. We can either have privacy or surveillance, not both.
>What if someone started an AI company that uses end-to-end encryption to make it impossible for anyone but you to access your data? Personally, I would switch to it in a heartbeat assuming it's competitive with the other products.
Why wait for a company to build it? Get your own local hardware like I did and have those guarantees because YOU set it up.
I don't consider investing $20k into hardware to run SOTA open models, that are far behind proprietary SOTA, to be competitive.
Even if open models were competitive, it's still typically going to be more expensive than a cloud provider because of low utilization and higher purchase price.
How do I know that a private LLM system won't have a degradation of quality similar to this or worse? The only thing I can think of for the proposed scenario is some sort of homomorphic encryption system? But not sure.
>How do I know that a private LLM system won't have a degradation of quality similar to this or worse?
You use benchmarks, you test, and because YOU'RE the sysadmin you know what weights are running at what time, it's very visible. You can airgap the hardware and be guaranteed it won't change over time. And, frankly, degradation over time doesn't seem to be what's happening with the open models.
Right now, I would be willing to pay ~$20 extra per month for strong privacy, assuming the same capabilities as Opus 5.5. I don't see local models making sense economically any time soon unless you value privacy at $1000's per month or are fine with much lower performance on hard tasks.
> Your follow-up about purely private tech seems to contradict your first statement.
That is correct! And exactly my point.
We want both, but, on paper, that is impossible.
But in reality, we make it sorta mostly happen anyway, through making the easy defaults not private, and the private stuff not easy.
This is not ideal, because [various reasons I do not need to tell you], but it has proven to be the best we can do to mostly achieve both goals.
Kinda like how capitalism isn't great but just the least worst option we've found so far.
___
The actual fundamental underlying problem being that not all people are equal, but we kinda have to pretend they are, because not doing so leads to fascism and other terrible stuff.
But we kinda also do not want to fully pretend that, because doing so leads to yet other terrible stuff.
Hence the quadruple-speak and contradictions to kinda sorta somehow have a somewhat functioning reality.
Oh don't worry, I'm certainly already on various lists, but the observations also will have resulted in the assessment that I am stable and no threat :)
I think we can have both and it might even be smart.
A lot of people causing issues are people that can't make sense of many things (like many terrorists). They get a fixed idea and they end up doing something bad. You would catch those with some (basic) surveillance.
A lot of normal people (not wanting to cause issues) might benefit from some privacy, if they understand what are the trade-offs (like government overreach). They can then use a slightly more complex tech.
We would still remain with the couple intelligent but sociopaths (think Unabomber style), but I think no solution can fix all cases.
That sounds adjacent to "I have nothing to hide". Everyone says that until they realize someone can change the rules. Before 2022, women didn't have anything to hide from their period-tracker app, now some have to worry about being charged with crimes.
Back to LLM chats: A system that can declare her "unstable" is also one that can permaban you from all air-travel because you "privately" said unflattering things about Dear Leader.
> That sounds adjacent to "I have nothing to hide"
I urge those people to share their full information from banks, companies (salaries, agreements, contracts), full health information and share publically all the texts they ever wrote (incl. as teenagers) and all the photos they have taken. And give away all the passwords to all of the services so people could check that they are truly clean. Just to be sure, just for everyone's security, right?..
The third highest voted comment wants surveillance. What is next? If you write "I'm going to kill that guy", which for non-autists means "that guy was really annoying" you should be reported.
In a old happy little idealised village, it became known quickly, who started to behave oddly and timely intervention could happen. In the modern anonymous mass cities?
No one (wants to) notice the madmen scheming in his isolated flat, surrounded by strangers. Until he explodes.
Unfortunately I also don't trust our government agencies with the surveillance - because they ain't transparent either and the self surveillance seems broken.
" "I'm going to kill that guy", which for non-autists means "that guy was really annoying" you should be reported."
And unfortunately there are lots of real threats being made under the disguise of humor. And much harder to separate im text. So maybe don't talk of murdering people in general, AI surveillance or not?
That is not my system, I prefer my happy village, thank you very much.
But in this reality, sure, rather surveil everyone with a baseline level and surveil dangerous ones gradually more. This is roughly how it works today .. just not very good. And with too many exceptions for the powerful.
> It should catch normal people becoming unstable so that they can receive help. It is just highly unfortunate that the US legal system works in ways where now this woman's name is public.
the honest actual opinion of the average consumer is probably that they don't want their chats to be monitored but they want everyone else's chats to be monitored. There was a lot of fury when a mass shooter recently used AI to help plan his assault. And of course there are all the people talking to chatbots about suicidal thoughts and intent. When they ultimately follow through, the providers are blamed for not alerting anyone.
No. This sentiment is why Snowden happened. We want privacy. Privacy isnt free just like freedom (whatever form it is) isnt free.
There are trade offs. ISP effectively is like driving on a highway, everyone can see where you are going but not what is inside the car. Id like these AI chats to be the same but they are not.
Devil's advocate: if I tell my therapist or my lawyer that I'm going to murder someone, they're obligated to report it. If I use my AI as a therapist or lawyer, why should the company that provides that service not be held to the same standard?
LLMs aren't email or file storage. AI labs aren't just shuttling bytes around, they're interpreting those bytes and taking action based on them. These models _already_ react viscerally in response to users saying disturbing things: the only practical difference is the ability (or obligation) for the model to escalate that concern. I'm not sure the ethics we hold AI companies to should be different than if a human being was typing out the responses.
Privacy is obviously hugely important, but this isn't the government surveiling every message. It's companies having an obligation to flag real, credible threats according to the law, which is a very different problem space.
Again, no. In your examples there is a real human there and not automated surveillance. There is a person who can be held accountable if they abuse that client confidentiality. Knowing big tech we will get a yt flock like automated system.
AI companies have no accountability as proven by the hugging face incident. And if they did, it would be the same old LLC non person taking ownership with a big slap on the wrist.
You are also wrong again that this literally is the gov surveying every message just like it is with them looking at your texts or google search. E2e encrypted messaging is hated because it cant be automatically surveyed. The idea it needs to be for law does not hold up as it doesnt for your human examples either.
I get the attempt here for nuance but still just, no.
People. Humans. We have got to have accountability. Once local llms are closer in capability to what people generally need and Apple has a local secure version of that, it is back to the e2e encryption and gov wanting backdoors arg again.
Almost all of these arguments boil back down to the “if you dont have anything to hide…” which id hope on hn we know is dangerous
>I guess we probably want our tech to work exactly like this.
Maybe you do; I want my tech to always include secure, encrypted communications. Don't include me in your destruction of privacy with your silly bandwagon!
uh no, 'we' absolutely do not want our tech to work exactly like this. Why would you assume that people default to 'search my anal cavity please' and not 'no, stay the fuck away from me with that glove'
----
jfc, why is this website full on psychopaths
"The actual fundamental underlying problem being that not all people are equal, but we kinda have to pretend they are, because not doing so leads to fascism and other terrible stuff.
But we kinda also do not want to fully pretend that, because doing so leads to yet other terrible stuff."
other people are not pretending everyone is equal. your power levels are showing, it isn't subtle.
True, most normal people have little concern for any such humanist goals and ideas.
Normal people live prejudice. It's (erm, claude-speak) load-bearing for them, given just how complex reality is and given just how well it reduces that complexity.
This kind of thing will get worse with humanoid robots because they have cameras and microphones. Will they be programmed to tell on you if you break any kind of rule in front of them because their owners are terrified of being sued?
this feels very thought-crimey to me, but I think I'd have to actually see that chats to really decide. Like is she making plans/asking for advice? is she just talking about her feelings exactly as if it's a diary?
This news article [1] had a quote supposedly from the chat session:
> The arrest report states that on Sept. 26 at approximately 5:10 a.m., Heller reportedly wrote, "I'm going to shoot up the sheriff's right the [expletive] now." The following day, at approximately 1:07 a.m., she was accused of posting, "This is 100% last chance I'm done. I got a new [expletive] gun today. [Expletive] you."
She is not being charged with planning, just making threats in a place a person could read - the person being employees of the company.
Basically, under this interpretation, any personal note you store in the servers of a company could qualify, even if you didn't ever imagine someone would read and as such you couldn't have thought about it as a threat
LLM is not a person but T&C probably states that a human moderator may view any of it. Her lawyer could probably argue a moderator filtering usage isn't the intent of the law, it was more about publishing / sending message for other humans and it was never clear to her that a human was reviewing her private diary. Shouldn't LLM disclose that at some point when people are feeding really personal stuff?
The law seems overly broad with "may be viewed by another person". Most of these laws have the intent of not causing public panic with regards to posts that others may see, not stuff that one assumed was private. This is further supported by the definition of threat, which is generally defined as requiring malicious intent.
I would think a good lawyer could get this charge dropped. I would like to see what judge approved the warrant and how they felt the elements were met.
This is practically entrapment. All the AI labs sure don’t shy from plastering annoying disclaimers everywhere saying their tool can make mistakes. Shame on them for not also reminding everyone continually that anything you submit can and will be used against you. Of course they can’t afford to have a Flock-style user revolt.
Not weighing in on the privacy issue but using Ollama you can run a smaller agent like Qwen 3.6 35b a3b on a sufficiently potent laptop. Pair it with something like Hermes for a nice interface and you have more than you might need for diary like usage.
These guys need to realise they aren’t the law, and they have no business with stuff like this. If I want to plan something bad I’ll use an ablated local model. Come on it’s sad to see how stupid the work is getting.
Could this be considered free speech? Most of the limitations I know of require you to say it to or in front of other people as to create a panic or make someone feel threatened. Is it a crime to say to yourself "Im going to blow up New York." or about threats of self-harm? How about "Im going to steal the Mona Lisa!"?
To me this is just straight up thought crime, they just don't have a way to directly read your thoughts yet. But they will spy on you and try and catch you out for it.
I downloaded a copy of a movie I had previously purchased yesterday. Claude helped me remux it so I could play it on my TV. Will the FBI be knocking on my door tomorrow?
Who knew the way to avoid charges for stuff like hacking the Australian government and other similar institutions was to report another person for a different crime.
> The communication must be made in a manner in which another person may view it
IANAL but it will be interesting to see how the legal system decides if this counts as "another person may view it" or not. What has happened in similar ish cases where someone writes a threat like that that they thing is private to them but actually ends up in the hands of Someone at some tech company that reports it?
I expect that anything I type / dictate / post / purchase on the web be it a chat conversation with an AI, a post on social media, a dm, a blog post, a blockchain reference, or an assett in a bucket, will be manually reviewed and forwarded to the authorities. It is their duty to do so, and I am glad that they do. If this woman did harm someone we would be hearing "why didn't Anthropic do something about this."
Aren't high trust societies staying that way by kicking out the incorrigible violators? Pondering violence might not be treated lightly even in a high-trust society.
Florida statute 836.10 puts the bar at the "sending, posting, or transmission of, a [...] record, in any manner in which it may be viewed by another person"
I think the defendant could successfully defend themselves by claiming they did not know (or intend!) the message could be viewed by another person, as they were plainly using it as a private diary.
The problem is that when using Apple Intelligence as a user, you never know when the OS will send your query to Apple Private Cloud instead of processing it with an on-device model. As far as I can tell, there is no way to keep Apple Intelligence enabled and at the same time disable automatically routing requests to APC if the OS deems the query is too complex to be processes locally. There simply isn't a single warning pop-up before your request gets sent to APC.
I can’t believe people are using these proprietary models/subscriptions as personal assistants, sending their most personal context, thoughts, documents and information to the providers. Some even give them full disk access.
After using their services for long enough, Anthropic/OpenAI essentially have your entire life mapped out to an insane level of detail, likely including detailed contents of your computer/phone.
People used to say that Facebook knew more about you than your closest friends/relatives. And compared to that, this is just on a completely different level.
Absolutely insane.
I'm sorry, but this is Anthropic being the volunteer thought police right? This woman is being charged with a felony for expressing a thought to Claude. Not because she tried to plan an attack with it.
It is like Anthropic wants Claud to be the pious antithesis of Grok. Although their ideals are far apart, the end result is they'll both violate your privacy if its suits their worldview.
It first started as something political but I really am starting to think Anthropic is misanthropic. The road to hell is paved with good intentions.
The guardrails, model downgrading, limited access, watermarking and now snitching on private thoughts (I know it's technically not but people perceive it to be so and act as such). It's all getting very dystopian very fast
After people getting pilloried for social media posts from twenty years ago, I really hope (sensible) people will have the wherewithal to think twice about what they hand over to their chatbots.
This is a classic case of "Damned if they did, Damned if they didn't". Given this has come out, and Anthropic is considered to be the beacon of transparency, it would be appropriate for them to share what their thresholds are. The article also reports Open AI not considering the threat credible enough to alert, so there's clearly some thresholding that people need to be aware of.
duck.ai et al? my chatbot claims its architecture provides some level of privacy yet it is honest enough to point out none of these systems are actually audited and "not stored" and "anonymous" does not mean "not monitored".
I haven't seen anybody claiming this for consumer/free accounts anywhere.
You get privacy if you're a big corporation that needs to make sure OpenAI/Google/Anthropic can't read your trade secrets etc.
But those contractual privacy protections have been in place for a long time. It doesn't have anything to do with AI, it's been the same with Office365, Google Docs, etc.
I feel like there is a double standard going on here.
There is another link talking about people generating images with AI that have signatures of artists. One, fairly reasonable, in my opinion, way to think about it is that the AI is not to blame for generating the image, but the person who generated it is to blame for publishing it. In other words, the AI service sharing the image with the user is different from the user sharing the image with the world because it's a tool.
But in this case, the situation is reversed: the user is sharing something with the AI. And then we see the hypocrisy. When it's the user doing it, the blame is found the instant it's shared between user and AI.
It's like AI companies invoke some sort of "user-AI privacy privilege" to shield themselves from criticism when the AI generates things people don't like, but that privilege only exists to shield the AI side, never the user side.
They'll enjoy the concept when it favors then, and pretend it never existed when it doesn't.
If it can be illegal to write something to the AI because it's considered "publishing," then it's equally illegal for the AI to reproduce something copyrighted to the user, as publishing has to go both ways.
O hey look that thing I and many others said will happen when there were all these public complaints about the tech companies not doing enough surveillance and tattling happened
Anyway, I'm gonna continue avoiding sending anything I can through servers controlled by a company - especially American tech companies - without some sort of end-to-end encryption with an intended recipient. I would highly recommend that every living person do the same. Obviously it's quite difficult to avoid completely, but given the capricious nature of both governments and the general public in what they're willing to come after people for, I'd rather take as few chances as possible
They still read every single message you send and train on them.
I’ve had them email me before because I was testing it as a filter for abusive messages and they detected some no-no and wrongthink in those test messages.
They run anti-abuse filters with ZDR but to my knowledge they don't have the specific prompts that triggered them. I have never received any emails from OpenRouter or otherwise.
I looked into ZDR, it's basically a vapid claim with little to no due diligence or auditing. I expect most providers to cave with only a minimal amount of legal pressure.
I find more and more parallells between AI and Google. Google searches have been used in court for over a decade, maybe two now?
The way I described AI when I first encountered it was "Google on steroids", and honestly it hasn't proven me much wrong. It takes all the results we used to find on Google, and generates code from it, so what? It's still just Google on steroids to me.
Don't google "how to hide a body", just as well as you shouldn't ask your AI how to hide a body. Not much has changed overall.
If it's going to be this binary. Then there has to be a shift in the way this is handled. It shouldn't be "You wrote these terrible things to a computer, you're now accused of doing them" to maybe "We where notified you wrote these terrible things, and the a psychiatric evaluation is being mandated by the state in which has a law about this sort of thing"
Obviously, as others have pointed out if they don't act and she does the crime it raises the damned if you do damned if you don't. But I've also had the AI's go haywire saying I'm doing all sorts of nefarious things when literally doing math proofs.
So no one size fits all. But going the extreme first is probably not ideal.
Claude did this, its also important to remember that a human counselor or psychologist will also do the same thing.
For example, if you said to your counselor or psychologist that John Smith is going to rob the bank next Thursday, they will report that to prevent the imminent risk of serious harm to others.
And remember that anything but local AI you personally control is not anonymous. Not even AI you pool with your friends.
Pre-crime prevention and detention will be the end goal. The economic devastation caused by AI will make large swaths of people get angry. Better use that massive amounts of data being hovered up and the mass of compute being built up to stop those angry people lomg before they even get a chance to properly organize.
They did what everyone wanted them to do: be proactive about insanity happening on their platform which in this case is mentally ill person wanting to attack a police station.
So now when everyone has their watch recording conversations, or perv glasses, or Alexas, or Jony Ives new personal AI gadget … recording everything we say and scanning it at scale … is it ok to be a doomer yet?
Tech companies have always done this. Don’t misread that as blanket support for it, but consider Anthropic’s position here. Do they want a headline to come out that says “woman used Claude to plan murder” or “Anthropic reported potentially dangerous person to the police”? AI being dangerous is already a hot topic, and this seems like a sensible move to me. If you want a guarantee of privacy, you’ll need to run your own models locally.
If you want technology that won't report you to the police for credible threats at they can easily detect perhaps create your own. I hear pen and paper work great for this.
Does anyone else remember when the tech-savvy crowd was wary of software "phoning home" or spying on users, or listening, or whether Gmail scanned and read your emails, and whatnot?
And here we all are, happily typing our stuff into these spy chatbots. "AI" happily made our fears go away. Hopefully we're not planning anything criminal like this woman, but still.
Snitching on the people - good mass surveillance company.
On the other hand, people need to learn to not trust these
companies. It reminds me of others being surprised when
a self-driving car reported a gun in the car. I mean,
do people not think? Besides, of course, it's already messed
up to want to have a gun. And it is constantly one country
that has such issues, more so than many other countries.
> Heller faces a charge of making a written threat of violence under Florida law. Florida Statute 836.10 makes it a second-degree felony to send, post, or transmit a written or electronic record threatening to kill or injure someone, carry out a mass shooting, or commit an act of terrorism. The communication must be made in a manner in which another person may view it.
Thought crimes are real when you're sharing your thoughts with Claude
I wonder if "criminal intent" may be missing here given that most people probably operate under the assumption that Anthropic are not reading their messages.
I think people have a binary understanding on this. Someone is either reading their messages or not. While the reality is that all the messages are read by a machine (not unlike GMail and Outlook) and anything suspicious gets flagged up so a human can read it. This obscure the concept of "reading" as most laypeople understand it.
Summary: don't type in Claude anything you wouldn't like a human to read.
It tells you exactly what it does with your information if you ask it, including this exact scenario, and has for at least four months now (when I asked).
Yes but you are someone (i assume because you are on HN) that at least understands this point. There are A LOT of people that use it as their confidant, expecting it to be private. There is a massive education issue going on as its not in the interest of these Corpos to make you fear sharing all your details with them. Because, then you wont get Dot or whatever Anthropic comes up with and share all your personal info with it.
You're not wrong, and it may come down to this in court, but there's a difference between what people think happens, or the reasonable expectations, and what actually happens, and I think it's important to recognise that difference and why it comes about.
I don't think someone is an idiot for thinking that the information they type into their private Claude account is private. I also don't think people are idiots for thinking their phone is listening to them and giving them targeted advertising based on that. Both are reasonable deductions from their lived experiences. Both are wrong.
I think it's a reasonable point to make. Writing things down is a part of "thought" for many, including those who keep diaries/journals. If you write in a private journal you do so with the expectation that is not shared, and that wouldn't seem to break this law (with my naive reading).
I mean, at this point it's pretty well known that all tech services will be hacked by fable, anthropic themselves promised it, so writing your journal in google docs or claude or a txt document on your laptop or such is the same as releasing it publicly yeah?
If it were written on paper, and only in a room with no phones or cameras so fable couldn't hack it, then I think you wouldn't be sharing it.
I think the zero-risk approach common nowadays is insanely corrosive to democracy and freedom. If a million people fantasise about shooting the sheriff, and one ever goes on to do it, I don't believe avoiding it warrants creating an apparatus of mass surveillance. After all, if people were really serious about zero murder, the only practical solution would be to lock up everyone. Some (most?) tradeoffs have exponential costs at the limit and we/lawmakers should recognise that.
> If a million people fantasise about shooting the sheriff
I think it’s fair to pre-identify folks who fantasise about shooting anyone. It’s a small fraction of the population that looks into logistics versus making offhand comments.
I don't have any number on murder, but rape is a fantasy shared by 55% of Americans, women more than men, many/most of whom necessarily thinking about its logistics in at least some detail without ever doing anything to anyone non-consenting. On murder, the popularity of true crime media means many people enjoy thinking about motives and logistics. I don't think it's a small fraction.
Why not report it? Seems reasonable. At the end of the day none of these services can guard against misuse. I personally optimized bomb creations and other stuff with AI just to see if it is possible. Maybe the way to secure the world is to not have corrupt old farts running every country. As the old saying goes. Information wants to be free. A kinder and more connected world is the only way to keep everyone safe.
This reminds me, for his short story "Dolan's Cadillac", Stephen King claims -- perhaps exaggerating a bit -- he did the real research on how to murder a person as described in the story. How to dig the hole, what measures, what the falling rate of a car that size would be, etc, etc. He claims he fuzzied the details a bit because he didn't want to teach readers how to actually do it (again, maybe he's exaggerating a bit to promote his story).
Imagine if, instead of a friend, he had asked an AI chatbot. Would he have been reported to the police?
Aren't our private thoughts just that, private?
I get that a cloud-hosted AI chatbot is, to tech-savvy people, immediately "not private", but a lot of people don't understand this. What if Stephen King was talking to his friend about planning the murder, and his phone was listening? What if he was typing it up in MS Word and the program decided to phone home and report him to the police?
> Another incident has taken place that illustrates the need to be careful what you tell AI
Nope! Another incident has taken place that illustrates how committed Anthropic and Anthropic AI is to the law and public safety, unlike all that open model riff-raff
They were waiting with bated breaths for something like this to jump on, and make an example of.
I have some sympathy for Anthropic here because I've seen the headlines after OpenAI failed to report a shooter in a similar situation. So from their perspective, it's damned-if-you-don't, damned-if-you-do.
However, people need to get it in their heads that they're not chatting with their secret BFF, they're chatting with Big Tech. Before LLMs, Big Tech had no way to scrutinize the bulk of what was going on within their services, so you could have a secret hate diary in Google Docs. Now, everything you say or write can be automatically screened for red flags on a planetary scale, and probably will be because that's what the regulators and "concerned citizens" will demand. In a couple of years, you'll be biting your tongue a lot more often in private chats.
Not just bad headlines; OpenAI is actively being sued [1] for this:
[1] https://cdn.arstechnica.net/wp-content/uploads/2026/09/Briti...In that case was the shooter using the LLM to plan the attack?
That could be a meaningful difference
That's what the suit alleges. Also that it had been automatically flagged to an internal human review team who had concluded that there was a credible, specific risk of harm to others and recommended notifying police, and that it was OpenAI leadership that rejected that recommendation.
You can see it all in GP's link. It's very readable.
... Wow. Well that's not at all what I thought happened.
I figured it just didn't get flagged properly.
So mentioning killing in passing is fine, but planning with the tool is not? How should LLM providers draw the line? I'd prefer if they didn't scrutinize any of my inputs to see if I am committing a felony or not in the first place - but that ship has sailed.
If it’s just a journal, how is it different then Google Docs?
...yes?
They draw the line using a team of humans who evaluate whether the threat is real and give a recommendation to management.
If you operate a gun store and someone reveals to you that they intend to use your product to shoot up a school you have a similar moral obligation, I think. This isn't a special case.
But it is, it somehow only applies to LLMs. What about doc apps? Todo lists? Chat apps?
There are tons of hosted app services where you can write private expressions. Why are only LLMs subject to monitoring?
You're really stretching here for some reason I can't fathom, but yes - if you write a credible threat to shoot up your school on a crumpled up paper and someone finds it, they should report it. I'm not sure that complicating this with which kind of app it's written in is all that useful ...
I don’t see how writing something you clearly intend to be private can be a threat.
Is this a point you're arguing seriously? If you find someone's plans to, for example, shoot up a school crumpled in a note on the ground that should be...ignored? Or what? I struggle to comprehend what you're actually arguing for here.
I think I pretty much agree with you that if someone finds evidence of danger to someone else's life then it is a fair expectation that they do something with that evidence. And despite that I still believe that is not a good enough reason to eliminate someone's right to privacy, and yet still that right doesn't mean that we should be able to demand privacy in all products. I think it is fairly clear that these big LLMs are not private products and we shouldn't expect them to be, but I'm still in support of offering private alternatives that people could use without having every input scrutinized.
I'm still puzzled that people's default assumption isn't that somebody is reading all of their internet communications. Ever since the Snowden revelations of 2013 I've more or less assumed that everything I type into a computer is stored in a government database somewhere. Not that I actually believe it is 100% of the time; there's a spectrum of trust, so I'm more confident that local apps on my Linux desktop are secure, somewhat confident in the end-to-end encryption of certain apps on my iPhone, but all bets are off for non-E2E encrypted data going across the internet.
I actually hope someone is reading it. It's so sad thinking I go to all this work to be clever and no one reads it.
Of course, if I don't want anyone reading it, I sure as heck don't put it online.
People not reading it is unfortunately pretty close to the truth.
You can bet that everything sent across the Internet is stored somewhere. But read? Yeah right. People don't seems to realize that there are 6B+ people online and just how big a number 6B is. If you have a staff of 30-40K people (like the FBI or NSA) and they spend 40 hours/week doing nothing but reading Internet posts and the average person spends 4 hours/week writing Internet posts (likely a huge underestimate) and the FBI agent can read 5x as fast as the person can post, then the collective manual surveillance capacity is about 2M people. That gives a 1 in 3000 chance that a random Internet user would be surveilled (though it is decidedly not random). With more realistic numbers of maybe 5000 analysts reviewing potential threats, spending 10 hours/week on it (so they have time to actually follow up on threats, attend meetings, communicate with their boss & coworkers, etc.), and the average person spending more like 25 hours/week posting on the Internet, that's a surveillance capacity of 10K people and your chances are more like 1 in 600K.
This was a reasonable argument until about 3 years ago.
I assume you're referring to AI?
AI will give you only what you prompt it for, and the prompts are still designed by human analysts. What's changed with it is that now the 10K people surveilled will be anyone that "fits" in a category that the government deems criminal (say being transgender, or foreign, or left-wing, or criticizing ICE) instead of actual criminals. It shifts targeting, not capacity. If you wanted to be an actual criminal and are willing to do it in creative ways that bear no resemblance to other major categories of criminality (or lets be real, political disfavor), there is no time better than the present.
Sorry mate. The one thing llms effectively guaranteed is that short of wonks like me, no one is actually reading the source. Everything is a summary of summary of a summary.
FWIW, I feel you; I too try to leave internet a mildly more amusing place to be.
reading what exactly?
For some, online discussion is just decentralized journaling
You just read some of it.
People don’t want to believe the Stasi is monitoring them. They want to believe the world is a nice place just like where they grew up.
Also, unlike the bad old days when 1 in 3 was an informant, now ordinary people aren’t in “informant loop” of providing information on others, so they aren’t thinking about being informed on either.
Or 'public/private information sharing' partnership ( hello Canadia ^^/ )
If we completely give up on assuming privacy, we lose even more privacy, because so much is based on the expectations of a "reasonable person".
Our devices, even Anthropic in this case, set an expectation that doesn't match reality. The software and devices we use are marketed and sold to us to be used for our private documents, photos, and conversations. I can't blame people for using them for their intended purpose. They shouldn't assume that someone is carefully watching everything they do and judging their every word and action looking for evidence of crimes. If we want people to think of the devices and services they bought and use as belonging to someone else and without any expectation for privacy we should really start with forcing companies to change how they market their products and services to reflect that reality.
> I'm more confident that local apps on my Linux desktop are secure
Why should you be? Linux gives us software we can reasonably say we own, but not hardware. Everything you type into your local linux apps can be being monitored by your processor and whatever is sitting in the CSME/PSP subsystem which you don't have permission to access, but which is always running even when your computer is off. We need fully open, documented, auditable hardware if allowing any third party access to our devices means our private documents will make us all into criminal suspects.
Most of the hardware isn't made in the US, so either the spooks would need agreements with every offshore business, intercept every hardware shipment, or convince every vendor operating in the US, even if they are not locally owned, to do this for all hardware sold here, and convince all of these offshore vendors not to ever leak the fact that it is happening.
On top of that, the data has to pass through several layers of routers and access points we control, which makes at least the fact of exfiltration visible. Plenty of people have a very strong vested interest in preventing data exfiltration from company hardware and so the government would have to somehow get in touch with every one of them to tell them to stay quiet before they leak any hints of it happening.
It's possible for targeted shipments, but the prospect of it being done for all consumer hardware sold in the US is a bit far fetched.
> Most of the hardware isn't made in the US
We know for a fact the government does intercept hardware shipments already on some scale. It's a lot easier when there are only a few chip makers being used for most devices. Intel and AMD alone cover the vast majority of the CPU market for desktops and laptops. It's basically the same situation for the wireless chipsets in every mobile device. No need to worry about dealing with about every cell phone manufacturer directly when you only need to hit up the extremely small number of companies every manufacturer has to get their chips from.
> Plenty of people have a very strong vested interest in preventing data exfiltration from company hardware and so the government would have to somehow get in touch with every one of them to tell them to stay quiet before they leak any hints of it happening.
Whistleblowers are extraordinarily rare. Edward Snowden worked with many many others. Any of them could have come forward at any time, but they didn't. Only one AT&T employee came forward to tell the public about Room 641A (https://en.wikipedia.org/wiki/Room_641A). The government marched into AT&T's building, took over part of their offices, rerouted their network and the whole AT&T backbone into their offices and through hardware. You can bet that more than one person noticed that. Government has no problems at all with going to a business with guns and gag orders and telling people to keep their mouths shut. Companies have no problems keeping quiet about what's happening either.
No one who isn't directly in the know is going to notice if someone's CPU (which is running it's own network stack) sends a few extra encrypted packets going to the servers of major internet companies (microsoft, cloudflare, apple, google) to either be collected at those end points or just picked up as it passes across the internet backbone.
Maybe if we had a ton more competition in things like chip makers, ISPs, operating systems, etc. it might be more difficult, but the way things are it'd be easy.
Yes, I mentioned the Tailored Access Operations NSA unit in a sibling comment. The thing is, open source hardware won't save you from any of this unless you have the manufacturing skills and equipment to build it yourself. Just like with open source software, if you aren't building it yourself, with your own hands, you can't be sure that what end result you have actually matches the code that you've read, unless there's a published checksum that you've also verified is real. Quite a bit harder to do with hardware than software though, especially since every single piece of the stack, including monitor cables, have been targeted by TAO.
Companies like Google and Microsoft pay top dollar to build special workstations to interact with production systems. These are not just special because of the software they run, but also the chips these machines are built with.
Sure, I'll take your comment at face value, not all consumer hardware has spyware, but it cannot be ruled out now, in the future, or at least now for targeted high value customers.
That’s… really not true or at least not as dramatic as that statement makes it sound.
Those “special workstations” are mostly about lack of features they view as potential attack vectors (external or internal motherboard ports, replaceable/non-soldered components, thunderbolt, anything extensible or “repairable”) and mandate certain hardware features (hardware tpm, locked secure boot, locked UEFI). Other than that, those workstations are Lenovo, Samsung, or Dell machines running typical chips. They may pick certain CPU models or chips but that’s mostly about avoiding new or “cool” features that may not have matured enough yet. Most of them are moving to thin clients though. Where you use that “special workstation” to remote into an VM that can access production systems. And that part is partially about controlling the software running in the VM and making sure updates can be forced “offline” on the VM even if you haven’t connected to it in few weeks.
> but it cannot be ruled out now, in the future, or at least now for targeted high value customers.
Well, sure, there was NSA's Tailored Access Operations unit which interfered with all kinds of hardware, even computer monitor cables:
https://www.nbcnews.com/tech/tech-news/report-nsa-intercepts...
For these kinds of operations, open source hardware won't save you unless you build it yourself, much like open source software. At the end of the day, you're hoping that the pre built thing matches what it claims to be.
The hardware case of TAO is still targeted and didn't rely on the manufacturer's compliance and secrecy for their entire line of products, though. The scale of that to my mind is quite the qualitative difference.
This is (seems) impossible.
You can have open and auditable hardware, but how can you be sure that the hardware you buy is exactly the same hardware as what's in the Github repo?
It's true that you'd have a hard time knowing a chip wasn't swapped someone on its way to you. There are a few things that would help but nothing easy. Getting the hardware into retail locations so that anyone could walk in off the street and buy a system with cash. Random spot checks, chip by chip. Packing the system in colored beads/rice to prevent tampering after it's in a box.
At this point, even if I could never be sure my system was secure, just knowing that systems were purchased, tested, and verified would make me feel a lot better. It'd be nice to have confirmation that it's even possible to make and sell a system that isn't backdoored in this country.
> I'm still puzzled that people's default assumption isn't that somebody is reading all of their internet communications.
I'm also puzzled by the phenomenon of using ones real name on the Internet (outside a professional context). I believe this began to occur around the time facebook became popular.
I've always used my real name, since I first got on to the internet in the early 90s. It's never been a problem. Big Tech and your own government know who you really are anyway, as do the big online advertisers. Pseudonyms are no defence against that. There is a potential risk from griefers, but they have ways and means as well.
> apps on my Linux desktop are secure
That's definitely not the good word to use. It is most likely made with the intent to be privacy-friendly, but they are unfortunately anything but secure (including the whole Linux userspace), and especially in the age of agents it would be the best if everyone understood it that you are a single bash/npm install/malicious PDF away from everything bad happening with your data. But rest assured, your video driver won't get updated!
https://xkcd.com/1200/
For technical people, this is incredibly old news.
For non-technical people, it isn't really news, because they already forgot about it after reading it. Maybe they'll be a little more monitored in their own typing for like... a day or two.
One of the hardest lessons to internalize, and keep internalized, as someone who works on and writes software, is the vast, vast, vast majority of the Public doesn't understand even the most basic shit about software. It just does stuff. Hopefully the stuff is good. That's it, beginning, middle, and end.
"Why would you think x would y" is a poor framing. They didn't think about x or y because they don't care. The phone works, that's the beginning and end of their interest in the subject.
You're completely right. I've internalized this on the technical side, but I still find it puzzling politically. Surveillance has been a salient issue in U.S. politics for almost 100 years now, since the invention of the telephone. I would like to think that most people are at least vaguely aware of Watergate, the Patriot Act, and Snowden. Then again, I should probably stop assuming that people know basic history, given the current state of education.
Honestly the older I get and the more I learn about criminals, of all stripes really be they petty thieves all the way up to state actors behind the most atrocious crimes against humanity we know of, they are all a bit on the dumb side. The ones that get caught are, anyway. Like so very often the perpetrator of a given crime just did one incredibly bone-headed thing that brought the cops directly to their door.
I think in part it's selection bias? Like if you're smart enough to get by honestly, you're probably also smart enough to realize getting by honestly is just a way more comfortable way to live. The only reason you'd probably cross that line is because your principals, whatever they may be, conflict with those laws, or your life circumstances are so bad that you have no choice BUT to turn to crime.
And that cuts the other way too: if you're dumb enough to think you'd NEVER get caught for a burglary, for example, you'd probably be way more down to plan and execute one, failing to consider that most thieves aren't caught when they steal the shit, they're caught when they try and sell it later.
Those in power have an interest in making criminals seem dumber than they are. Those who get away with crimes aren't part of the calculus either.
Those in power are often dumb criminals themselves. The fact that the system will break its back for an excuse to not prosecute them doesn’t make them skilled operators.
> Honestly the older I get and the more I learn about criminals, of all stripes really be they petty thieves all the way up to state actors behind the most atrocious crimes against humanity we know of, they are all a bit on the dumb side. The ones that get caught are, anyway.
...but the last sentence if kind of important, right?
We only know about the criminals who got caught. Which probably means they did something dumb.
It's conceivable to me that there could be many—maybe even a majority—of criminals we don't know about because no one ever caught them. Maybe they committed fraud once, decided not to push their luck, and lived the rest of their life quietly. Maybe they killed someone and made it look like an accident, and no one ever suspected anything. How would we know?
You can look up crime clearance rates. It's a little depressing.
Here's an FBI chart for 2019 [1]
Though some countries like Japan and Germany have really good clearance rates for homicide. [2]
But more heartening, it seems like it doesn't really matter how many people the cops catch. Crime rates are still generally on a long downward trend (with some bumps along the way). [3]
[1] https://ucr.fbi.gov/crime-in-the-u.s/2019/crime-in-the-u.s.-...
[2] https://en.wikipedia.org/wiki/Crime_clearance_rate
[3] https://ourworldindata.org/grapher/types-violent-crime-rate-...
And even those are only the crimes we know about! I imagine a really smart criminal wouldn't let anyone know a crime had taken place.
Zodiac killer was never caught and it seems he already died a few years back - it was established some time ago through some teen's DNA whose grandfather was ex-cop or something. Don't quote me on that but it's on the net somewhere and was there before AI slop happened.
Basically a lot more evil goes unpunished than most realize, because carma is a thing only when it's a thing - sometimes it just isn't.
I actually completely agree with that. I guess it depends how you want to slice that in terms of categorization.
Which is to say: If a criminal gets away with a crime and faces no punishment, not even public scorn, are they really a criminal? Or did they just work a situation, even of their making, to their advantage?
I don't particularly know where we should draw that line. But yes, my statement does deserve a big caveat there.
> getting by honestly is just a way more comfortable
I think it's also much about self-esteem -- am I proud of myself? And criminals probably think highly of themselves when they succeed in their crimes.
It isn’t that black and white. I have relatives that are highly non-technical but still care a lot about privacy and data protection in their computer use. They hate having to deal with technical things and have very little technical understanding, but still ask me whether using X software in Y way might pose privacy issues or not.
TV is a human in the box. A phone is internet in the box. It's all implicitly local, because you clutch this internet in the box right in your hand. Synchronization sends data directly from one internet in the box to another internet in the box.
I wouldn't be surprised if a dedicated enough IT at a company is not monitoring your enrolled devices microphones. Just listening in the background for shit-talking the company.
It is funny that you mention that, because at home I do/hear/watch all sorts of things, so I suspect transcripts look fun and may make management feel a little icky ( makes me wonder ). Honestly, one would think it is a reasonable position that they would not want to know too much:D
I understood this a while back when seeing that hedge funds etc require employees to lock away their devices.
no outside device comes in - no inside device goes out.
You know, at my previous job the company had a semi-strict policies with non-enrolled devices. If we didn't enroll our devices we were locked out of some company systems, but we could still use email and teams (which is like 99% of my use-cases for my phone).
So I obviously opted to not enroll my phone, but that came to bite me in the ass one day when I needed to get a train ticket from my company email (it was a business trip) and the policy prevented non-enrolled devices from attaching or downloading attachments...
Felt like a fool grabbing my laptop to let the train attendant scan the QR code.
Worth it still. Next time print the ticket on dead trees?
funnily enough, afterwards I would just forward it to my personal email. IT never bothered me about sending .pdf attachments from my company email to my personal email...
I'm still puzzled that people's default assumption isn't that somebody is reading all of their internet communications
Wow, so we're assuming police state plus it's your fault if you didn't know...
and this assumption is wrong how exactly when viewed through the lens of real-world events?
I'm not talking about the lens of morality nor even the law, because it's obvious this shouldn't be the case but in actuality, it is - everyday more true than it was yesterday.
I don’t think Google is reading all of your email in this way, or your phone text messages being read in this way, or say iCloud or OneDrive files being monitored in this way.
Certainly it shows that agents like Muse are a complete nonstarter for anyone doing anything that needs to be private. Even if you never talk about it with the bot, merely having sensitive information on your hard drive means it can be sent to outside servers where, if it hits some safety filter, some probably low paid employee is going to read it. Say you’re working for a public company and have files on your computer that could be material nonpublic information, now potentially some content monitor is going to be seeing that and trading based on it. It’s completely untenable to have everything you put on your computer be subject to human review at some tech company.
All your communications which are available are being swept up by efforts like prism and stored.
I think you can count on governments accessing that information where it suits them, and you can definitely count on unscrupulous people like Altman within tech companies to do so too.
I’m not sure what the answer is here, but it’s naive to think that nobody is accessing the information you give away by sending it to cloud AI providers.
Everything you're saying here is correct. It operates on the power of naivety.
Anytime the strong encryption debate comes up it's actually about maintaining the effectiveness of unencrypted surveillance in the AI era.
This is one of the reasons why AI companies are looking for explicit regulations, it can help to reduce the risk of possible liability and maybe make the legal way forward more tractable. With a regulatory framework in place much of the burden of identifying risks falls on the regulatory authority.
Then the AI companies have more confidence that they can move forward in a certain way, and issue investor guidance that is maybe closer to reality.
I think they're mostly looking for regulations because they've spent close to $2T, all to realized they have no technical moat, so they're trying to build a regulatory one.
Both things can be true.
They want stable rules they can follow, which will limit their liability as long as they stay within them.
They also would like those rules to be as restrictive as possible towards their competitors.
They've spent close to $2T so far. At that scale the legal issues they've had are just the cost of doing business.
I'm quite sure that if there wasn't the existential threat of a lack of a moat, they would not be pushing for regulations at all.
The legal issues can sink their entire business model, not quite the cost of doing business.
A lot of things could sink their business model including opening the can of worms of the wrong regulations.
After all the claims of "the product we are developing could cause the extinction of humanity" I'm honestly surprised Florida is the only place so far to seek an injunction preventing further development of the product.
It's really hard to take these companies seriously or at face value about anything they say.
As restrictive as possible towards their competitors, while being as permissive as possible for them. Getting control of the regulation making process means they can make sure they don't end up too constrained anywhere that would hurt their bottom line.
That's the conspiracy theory, but there's no good evidence for it, and it doesn't really make sense in the long-run.
It's the only thing that makes sense for a lot of these companies to survive to any long term when open models keep nipping at their heels for pennies on the dollar.
You're assuming the open models will follow regulations though.
No, I'm assuming that the open models won't/can't and will be banned from a lot tons of use cases that will mandate use of the large frontier shops in order to comply with said regulations.
The theory is that open models cannot follow regulations and will therefore be banned or not eligible for many of the large contracts that the frontier labs will win.
No good evidence other than their openly stated desire for that regulation? Dario and Sam have both made open pleas for it, repeatedly.
Liability is enough of a reason
And such was the case for a man who snapped a photo of his own child to send to the doctor which got uploaded to his Google photos resulting in his Google account of over a decade getting shutdown for CSAM.
As another commenter said, you're not chatting with a friend; you're chatting with Big Tech.
How much do you love Big Brother?
Wasn't this because the photo was made "public" as in, shared through Google Photos?
If the doctor was not using a gmail account, the UI probably recomended to share it "with anyone that has the link" that is like public but protected by oscurity.
Most people don't realize that it is 99% like posting it on Facebook.
It isn't like posting it on Facebook. It is like emailing it, because most people don't have the link.
If it was a password entered into a form would you still consider it protected by obscurity? Both are the same level of security.
The URL may be logged by the ISP, university/company or whoever is providing the connection, antivirus, windows recall, browser history... But IIRC passwords are send using cryptography if the site uses https.
Its nothing like posting it on Facebook. Google doesn't have a recommendation engine that tries to share the image with random people, or a search that image will appear in, or a set of subscribed people who will be notified.
Its a random link someone would have to guess.
I can't read past what the paywall allows, but here's the nyt article: https://www.nytimes.com/2022/08/21/technology/google-surveil...
> people need to get it in their heads that they're not chatting with their secret BFF, they're chatting with Big Tech
Yup. And with zero privacy protections in statute for AI chat, there is nothing to prevent an AI CEO looking to curry political favour from e.g. handing over the private correspondence of an opponent or an entire district’s residents.
Or something like leaking every non-corporate ChatGPT user's chat logs (including temporary chats) to the NY Times.
Or making up stuff
Don't know why this got downvoted? This is a problem with epistemics.
It's more efficient to have one central "verifier" for everything, but the "who watches the watchers"? question basically says: Either constrain by construction, have everyone verify (which are two sides of the same coin, btw, when looking at a "global" thing), or centralize explicitly.
Customers who what privacy protections for AI chats are welcome to negotiate this in enterprise contracts. The major LLM vendors do offer that as an option. Customers can then enforce any violations in civil court (although this obviously wouldn't apply if the customer used the LLM for criminal purposes).
> welcome to negotiate this in enterprise contracts
I really hope you mean that in an insulting way to the industry and current legal situation, not as an actual solution.
Where's the insult? Lots of customers have such agreements.
That anyone smaller than an enterprise gets screwed. It's like responding to a problem with "let your butler handle it".
Individuals can sign up for enterprise agreements. An enterprise can consist of one person. Or run a local model.
It seems like some people seem to think that a chatbot should guarantee total privacy like a confession to a Catholic priest or something. That is ludicrous and unrealistic. It's a commercial service subject to terms and conditions. If you don't like it then don't use it.
> Individuals can sign up for enterprise agreements. An enterprise can consist of one person.
Custom negotiation is a big difficulty for individuals and the pricing will probably have fixed overhead that becomes pretty ridiculous when it's not spread over a hundred or thousand users.
> Or run a local model.
Much better advice. But still an obnoxious amount of effort and restriction to avoid spying.
> It seems like some people seem to think that a chatbot should guarantee total privacy like a confession to a Catholic priest or something. That is ludicrous and unrealistic. It's a commercial service subject to terms and conditions. If you don't like it then don't use it.
I want it to have the same privacy as using a local program. Does it still sound ludicrous if I put it that way? I'm not talking to a person that would have to suppress their own conversations later, and there's no need for any data from this chat to escape into the outside world. It's like wanting a phone line where the phone company won't listen in; even if doing that is standard procedure and I can walk away it's still gross.
In the case that the company running this program for me is served a warrant, I accept that saved data would be shared. But proactively screening my data for a reason that isn't to help me is bad. We should fight to make that easier to avoid.
Also you're talking out both sides of your mouth when you say it's "ludicrous and unrealistic" right after telling me I can negotiate that exact feature in an enterprise contract.
Yes, it still sounds ludicrous if you put it that way. If you want a contractual agreement for privacy then pay for it. Everything is available — for a price. But a civil agreement can't bind a counterparty to conceal evidence of a crime.
Privacy should be an offer from the start if it's not mandatory. It shouldn't need special negotiation.
> But a civil agreement can't bind a counterparty to conceal evidence of a crime.
Are you talking about a different scenario? In a chatbot situation, the crime of "threat" requires snooping to even be possible. If the data is private there is no crime to begin with.
Wow. Now that is a weird and interesting take. Paying for privacy. Wow. I guess next we will be paying for respect too.
this thread is about ordinary people using chatbots for personal use, not enterprises for work
I have expanded the scope of the thread to make it more sensible.
The more sensible thing would be to link to any example of any lab anywhere offering enterprise solutions—with the opportunity to negotiate—to individuals.
People are confused at your take because it’s based on a reality that so far you’ve only described as the better/obvious option for individuals. Companies treat corp customers and individual customers differently. This isn’t new.
My company did negotiate the data policy amongst other things for our enterprise plan, I was involved in the process. I can say with certainty that they’re not picking up the phone for _one dude_ who wants the same thing.
I suspect it will go further: imagine giving an mp3 to LLM to clean up some noise. It detects it was illegally downloaded from youtube, deletes it and automatically fines you via attached credit card.
Perhaps but no indication of that so far. Agents are happy to set up *arr stacks today.
We have weaker versions of this already.
I had to move some games from C:\Games\RandomWarezGroup\GameName to my Steam directory, because Claude refused to reverse engineer an asset format due to piracy being incredibly immoral and illegal and blah blah.
They could be happy to assist you with a task that they also then flag and forward to authorities, especially if their assistance doesn't break the law but has evidence of you doing so.
Sure, and they could be a front for space aliens choosing who to abduct. Anything is possible!
You mean forward to the copyright holder or more likely a copyright troll who paid the copyright holder for the right to represent him as lawyer.
I'll be glad for open models when the day (inevitably) comes that the proprietary LLMs no longer work in my interests.
> LLMs no longer work in my interests
As articles like these show, cloud-based LLMs don't work in your interest today.
Is it in someone's best interest to let them die in a shootout?
Level 1:
Have you ever thought of doing something you wouldn't be proud of, and then not actually acted on that thought?
If so, or if you can imagine a journal or always-listening assistant hearing your muttering thought. Should you be charged with a crime for that? Who is the victim and what are the damages?
Level 2, the slippery slope:
Suppose you travel to a country where your sexual orientation is illegal. If your AI company or social media provider knows your orientation, should they be compelled to reveal your crime? Why or why not?
> Have you ever thought of doing something you wouldn't be proud of, and then not actually acted on that thought?
This - specifically with a shooting, is basically the plot of The Drama (film) from earlier this year.
No AI in the movie plot though.
That it's a tough line to define doesn't mean there isn't a line, unfortunately.
For them to die in the shootout, you have to assume there was actual intent to execute said shooting.
"According to the arrest report, a user identified as Carli made a statement on Sept. 26 saying she was going to 'shoot up' the Lee County Sheriff’s Office. Investigators say the same user made another statement the following day saying she had gotten a new gun."
At a certain point, intervention seems appropriate. This meets my personal threshold, and I'd hope a friend would do something if they came across that in my journal.
Unless it becomes a requirement to be licensed to be able to use any kind of ai model. You know, for safety and stuff. And of course with appropriate reporting to institutions.
There’s no recognized inalienable right to keep and bear models/GPUs.
I heard an ad on the radio this morning from the local Internet support/WordPress/website developer/seo company that they've moved from seo into astroturfing for _LLM_ input. They advertised that they'll make sure you're company is the one that gets returned and trusted when other companies search the web via agent. If this works for agentic search, it probably works for _training_ as well, so good luck finding an open LLM that's been trained to follow your interests. Has there been any research into whether/how much LLMs are more likely to recommend companies whose names they saw in training over similar but smaller or newer command also returned in search results? Given how well advertising works on people, I can't imagine that it _doesn't_ work on LLMs too.
Ask it how to evade astroturfing.
That day was yesterday. LLMs from big tech regularly refuse to do what you want.
Wouldn't they have to fine themselves first?
They were fined for their illegal downloading. More than a credit card limit.
And dramatically lower than their expected value from the copyrighted material they scraped
Or worse: downloading a picture of pirate ship and without any concern for the copyright asking the LLM to make a coloring page for your kid. BTW Chatgpt does that way better than Claude
Hey but you’ll be allowed to file a response that will also go to an LLM and deny you automatically. And you will be charge a NSE fee (no sufficient explanation).
I think this is the only business model that makes sense, monetarily, for current LLM CapEx and OpEx. The value is not in what it can do for consumers or how much money it makes, but how it can shape consumers through knowledge control. Think of how much a totalitarian government would pay to keep citizens from thinking certain things.
I don't even think government would be the primary customer. Advertisement is already that.
> people need to get it in their heads that they're not chatting with their secret BFF
I see constant ads on video platform (particularly youtube/tiktok) about llm chat apps, from friends, dating, romance and everythkng inbetween; that's personal.
People need to be reminded constantly if they use such apps that they are participating in easier mass surveillance, profiling and AI training.
Problem is that even not using those apps, the apps you currently use might have turned more hostile.
It wasn't technically feasible to scan personal chats easily, other than grepping keywords which must have had a bajillion false positives. Now you can get everything autoscanned at scale.
How do you get that through to someone who doesn't even understand that mass surveillance and profiling is a problem? Or to young people who have only lived in a society of mass surveillance?
The only way is for them to suffer consequences of mass surveillance.
But then we have to as well ={
A disclaimer on the top of page every time would have been a better approach helping Anthropic and the end user.
A bot talking to you directly as if its some one real caters to your thoughts and can take you in a certain direction without you realizing it. I have heard first hand experience from people that they feel more comfortable talking to chatgpt or claude cause it gives a feeling of being on their side and listening to them.
I'm hoping and actually expecting that eventually there will be a simple way to locally host a small model that is good enough for simple things. For a lot of use cases, the capability of free chatgpt one year ago was more than enough.
Maybe there already is! But last I checked it was a little bit of a mess of manually installing things from multiple websites with little documentation.
You should probably get a head start on waiting a couple years to bite your tongue and assume everything you type into a computer is summarized and sent to your boss, government, advertisers, political actors, insurance companies, worst enemy, etc. With phones, Alexas, and little AI tamagotchis, you probably shouldn't say much in person either.
The Silent Generation, version 2.
They're actively rummaging through your inputs so the damned-if-you-don't case doesn't really exist; no one expects Anthropic to not notify law enforcement once they learn of something like this. What you might have expected was some privacy in the first place though, where Anthropic would never have learned of this in the first place and where the damned-if-you-do case wasn't a thing.
I'd expect someone's BFF to call the police if their friend is making credible talk about shooting up a school or whatever. I sometimes feel like privacy talk online can be seriously divorced from how human relationships actually work.
Sure but you wouldn’t expect your filling cabinet to call the cops on you. Because your filling cabinet doesn’t read your papers. The problem is the spying not the acting on credible threats of violence.
> Now, everything you say or write can be automatically screened for red flags
Considering that many people will share their deepest thought with LLM, it might start looking more and more like attempts of precog agents from Minority Report...
Not sure why so much sympathy for any company that makes its bed and now has to lie in it.
> people need to get it in their heads that they're not chatting with their secret BFF, they're chatting with Big Tech
It's the other way around, big techs need to properly disclose in their platform, during interaction that they aren't in a private and safe environment
This is something of a HN trope, "I'm sympathetic to X 'cause they tried to do their thing one way and people complained so now it's OK if they do it the opposite way, even if that's kind of F'd up too".
The problem with that is, "no, they don't have to do their thing. They have no given right to do it. If they do it, they should, they have an ethical duty to, do it right..."
> in private chats.
Including any chats anywhere where someone might have a phone in their pocket, or if there's a "camera" attached to a utility pole or a nearby tree. The only real private chats might be whispered lying down in the bathtub together, with a mattress covering it like you're both hiding from a hurricane.
> they're chatting with Big Tech
They're chatting with any powerful person who wants to hear it. She thought she was chatting with Anthropic, who doesn't give a shit about her. But after being threatened (and immediately backing down because, of course, they don't give a shit about her) Anthropic has become an arm of the government. So she was chatting with the Bonita Springs, FL Sheriff's office, or anybody else. If I paid enough, Anthropic would tell me about what she was doing so I could sell her laundry detergent.
Right, the difficulty is partly that they can get a negative headline from any choice of behavior.
"Anthropic failed to report murderer's threats to authorities"
(or "Chatbot knew man was planning murder, yet company did nothing")
"Anthropic reported private chats to authorities"
(or "Arrested for chatbot fantasy")
To be fair to the journalists in these cases, there's also no society-wide agreed Schelling point about the correct outcome or correct rules. I have strong beliefs and intuitions about what should happen, but other people also have strong beliefs and intuitions, and many of those are probably opposite of mine. Even if my intuitions are the best and most justified, a journalist is unlikely to think "I'm just not going to mention that some people are mad at this company over this outcome, because a hypothetically better norm or principle would support the company's actions here". Hopefully the journalism can at least contextualize the lack of legal or social consensus and the difficult incentive problems, rather than jumping to "obviously companies are sociopaths staffed by supervillains".
It's a byproduct of the nannyism safety marketing from the AI companies. I'm glad these cases were caught, but disagree with how they were disposed of. If the automated flagging is good, enforce it by default. If it's noisy, refine the tech then enforce it by default. This middleground where everything going through the platforms is subject to training and arbitrary human inspection in the midst of an acrid cloud of marketing-driven fearmongering is unacceptable, and it reinforces the idea the fearmongering is legitimate.
Somehow humanity survived the past 40 years without Microsoft Word and Excel phoning home and shopping users to the feds at random, I don't see why the standard should be any different for this new class of tooling.
As if it is just nannyism marketing by tech companies. The EU tries to bring a new nanny law into legislation every other month
> they're not chatting with their secret BFF
If a person told their real BFF they were planning a shooting and went through with it, couldn’t the BFF be charged with accessory?
I don’t think so under historical English common law. If you lift a finger to help, i think so. If you see everything with no aid whatsoever, I don’t think so
Curiously predates US law, it’s old English common law still operating as a principle
IANAL blahblah
> Before LLMs, Big Tech had no way to scrutinize the bulk of what was going on within their services, so you could have a secret hate diary in Google Docs.
They did and they do
https://www.bbc.co.uk/news/technology-44699263
I never considered online services safe. What I object to is active scanning of content on devices. This should not be allowed.
Yeah. Damned if you do, damned if you don‘t.
This isn't normal, this isn't how technology has worked.
Google doesn't report people to the police for the private (non-CSAM) contents of searches or emails.
You could definitely go through people's shit before, but Big Tech generally had serious prohibitions and a stronger presumption of privacy about this sort of thing.
CSAM was an exception where they theoretically had the evidence of the actual crime, rather than writings alluding to them. And even that has been problematic.
Frankly, the charge here is total bullshit and should be thrown out. The law is meant to prevent people sending threats to others, not keeping notes. If this charge sticks, the law should be changed.
it was in the article
Absolutely no sympathy. Anthropic is every bit as slimy as any other big corp. And like other big corps, they must open the vault to whatever governments they intent to do biz with.
No sympathy for whom?
For Anthropic? They don't need it.
For the woman, whose entire crime was using a chatbot that runs on Anthropic's server, and wouldn't be charged or guilty of any crime if she used a similar chatbot on her own machine to write the same exact thing?
The woman whose only crime was using the cloud to store private information (diary entry, stream of consciousness, thought experiment, etc) assuming it will stay private, as is normally the case?
The woman who was jailed on a technicality that allows the government to treat the 1:1 conversation with a robot as a public threat because the fine print says that the corporate has the right to snoop on the conversation?
I hope the the woman gets some sympathy from whoever reads this news, because it's a travesty and perversion of justice that this took place and Anthropic assisted in.
This crap makes KGB look good.
We really need a class, probably in high school, that works through how LLMs work at the high level (don't need to get too far into the deep math, but give people a taste) and then how they're trained, used, and deployed.
I feel like if people understood what these things actually are there'd be way less of this AI psychosis and similar stuff.
There'd also be fewer people falling for apocalyptic Rationalist delusions.
Also: people need to understand "not your computer, not your data." (Unless it's stored in the cloud but encrypted locally with keys only you possess.) Same goes for storing things unencrypted in OneDrive, Google Drive, etc. There is nothing to stop these companies from bulk scanning, data mining, or reporting people based on whatever request a government gives them. Don't count on them to resist, because they often can't, especially if the request is from a sovereign state where they do business.
You can make a reasonable case for adding a lot more classes in high school: statistics, nutrition, personal finance, etc. But ultimately it's a zero-sum game and to add a new class means removing an existing class. So what do we cut?
My kids' high school covers those topics. And I think that is not a new thing, I specifically remember taking the required personal finance class 35 years ago (great teacher -- crotchety old man who wrote "compound interest" on the blackboard every single day before class and repeatedly proclaimed that if we forgot everything else he taught, please try to remember how compound interest works).
There'd also be fewer people falling for apocalyptic Rationalist delusions
Assuming you consider it a "delusion" to have a p(doom) of more than 5% or so, that's not uncommon among frontier lab employees who have a pretty good idea of how LLMs work.
Indeed, I don't see how knowing the details of how LLMs work (which I know btw) would change anything about how intelligent they are. I only need to know that it's a computer program that writes stories, solves math problems and seduces people.
> There'd also be fewer people falling for apocalyptic Rationalist delusions.
Um, this is coming from the researchers building these models. So, good luck?
>We really need a class, probably in high school, that works through how LLMs work at the high level (don't need to get too far into the deep math, but give people a taste) and then how they're trained, used, and deployed
Not applicable.
If the woman was chatting to a local LLM, there would be no way to charge her for a goddamn thought crime.
Which is what the government did here, with Anthropic's assistance.
Thinking about committing a crime isn't a crime.
As for treating the "threat" as "public".. something tells me if the court ordered to share something publicly, telling it to a chatbot would not count.
>Also: people need to understand "not your computer, not your data"
Rrrright, because it's so easy for people to know when stuff from their computer is transmitted to not their computer.
There's nothing physically stopping those companies from scanning data on your computer either.
And legally, surely the state that treats convo with a chatbot as public when it suits them would prosecute mega corps for overly thorough telemetry when it also suits them, right?
Why people here accept this as normal is beyond me.
>I feel like if people understood what these things actually are there'd be way less of this AI psychosis and similar stuff.
There are people with education here that don't fare much better, so I don't know.
> I have some sympathy for Anthropic here [...] So from their perspective, it's damned-if-you-don't, damned-if-you-do.
On the other hand, they did put themselves into this position deliberately.
By offering a product?
Show me any product, no matter how simple, that has no safety vs utility tradeoff.
They have some responsibility for people's expectations of the product, at least. They want the personal assistant personas to be able to help you with anything, and don't point out that they'll be judging your thoughts along the way.
For anyone technically inclined it should be obvious, but it isn't part of the zeitgeist or how they pitch it. People see it as being different than talking to a human, and behave as if there won't be a human in the mix.
>By offering a product?
Yes.
>Show me any product, no matter how simple, that has no safety vs utility tradeoff.
Show me a product that was not offered willingly.
Observe that Claude isn't one.
that tells users upfront "we can read whatever you type in here"
(and not buried on paragraph 56 of a 20 page TOS that no one reads or understands)
You would first have to demonstrate the utility of whatever product Anthropic is offering.
Pool together with some friends and buy an H200 or two to run unquantized open source models with abliteration/heretic transformations.
You need to be able to use these models for the real world and not for some imaginary world where everything is safe and nice and happy all the time, while at the same time intensely surveilled in the name of CYA and the latest panic about whether speech THAT ISN'T EVEN BETWEEN TWO PARTIES is considered "wrong".
I'm a free speech fan that acknowledges there are lots of boundaries of free speech (fraud, perjury, blackmail, defamation), but the one thing that all of the boundaries have in common is that a second party must be involved for them to make any sense at all.
Maybe the courts will uphold this, maybe they won't, but don't take the risk!
Once you go in on infrastructure you have become a small data center. You will need to maintain it, continuously finance it, and secure it. And what if someone wants to back out 6 months later? Now you’re signing contracts with the implication that you’re willing to take a friend to court.
You can get around this by hosting in a 3rd party data center, but now you have the same trust issue again, but with more steps.
It’s all overkill for most people anyways IMO. This lady was just using it as a personal journal. Basically a glorified ELIZA. That kind of thing can be done with really small models locally these days.
> This lady was just using it as a personal journal. Basically a glorified ELIZA. That kind of thing can be done with really small models locally these days.
And she's facing felony charges. Yeah, folks on hacker news can talk about technical solutions all day long, but that's not the fundamental problem. The fundamental problem is that we've allowed our digital infrastructure that includes everything from banks to schools to doctors to become adversarial to us. In some cases predatory.
It's effectively untrustworthy, like being charged a felony for walking across a bridge incorrectly. That's not a "I'll host a small model in my garage" problem. It's a we need the government to do its job problem.
Well this is why I think small local models are the opportunity of a generation. Because it’s a bit transgressive these days to say you don’t need a big centralized service that spies on you and requires a subscription. It’s not the absolute best, but it’s yours.
I agree in general that we technical solutions to social/political problems are suboptimal. But technical solutions don’t require the political system to work, which is a very attractive property at a time when politics seems broken.
There is a counterpoint that retreating into technical solutions only cedes more ground and makes things worse, and for that I don’t have a good reply other than the fact that life is short.
I mean, this is also just kind of how mandatory reporting works.
If you tell a teacher, a therapist or a priest that you are going to kill somebody, in many states they do actually have to report that and can be held liable for the resulting crime if they do not.
Every time we have a mass shooting in the US it inevitably comes up that they wrote it down or told somebody and then the next logical question becomes “how could we have stopped a mass casualty event?”
>the next logical question becomes “how could we have stopped a mass casualty event?”
'No Way to Prevent This,' Says Only Nation Where This Regularly Happens
But ultimately it’s not a technology problem it’s a society problem
The people who are accountable and responsible are delegating their authorities to policy and now the policies are just implemented by machine systems and there’s decreasing human intermediation.
The humans that are still inside the system increasingly have less control such that it’s it’s increasingly difficult to find anybody who is the actual customer service type representative who has the authority to make a meaningful impact.
We need actual privacy laws in the united states. Between this, other cloud providers, the advertising tracking infrastructure, phones that constantly ping your location, flock and it's ilk, etc. It's over, no longer can you live semi regularly and still maintain your privacy by maybe living out in the sticks a bit. This is the panopticon.
> You can get around this by hosting in a 3rd party data center, but now you have the same trust issue again, but with more steps
No this is not true — colos are very hands off.
You don't need to operate a mini AWS to self-host, and I don't know why people on HN want to make it sound like self-hosting if you don't have nation state security
> colos are very hands off
That’s a pretty broad statement? Maybe a more correct one would be that some are hands off enough that some people might find them worth it.
And that really only addresses one problem. The other one is the coordination problem. You go in on a $30k GPU with 3 other people. So each person puts down 7.5k. But then one person wants to pull out, and so every remaining person needs to put in an extra $2.5k on top, and they’re questioning whether $10k for a chatbot is really worth it.
This level of hardware and cost just isn't necessary. You don't need to team up with your friends to go all in on a hardware purchase, which I've never heard of anyone doing anyway. Go spend some time over at r/LocalLLaMa, you'll see.
I've never heard of a colo looking into their customers servers. They're not in the business of monitoring their clients.
I don't understand the coordination problem – if you want to pull out you have to find someone else to buy your share?
I've seen something like it once, but it was really just the FBI coming in a seizing a rack.
> Once you go in on infrastructure you have become a small data center.
A PC at home with a 3090 in it is more than enough if you want to talk to a chatbot about your day.
> Once you go in on infrastructure you have become a small data center.
Wow. I never knew that my Frankenrouter, gaming PC, and handful of laptops, switches, and WiFi APs were a datacenter. The things you learn...
> This lady was just using it as a personal journal. ... That kind of thing can be done with really small models locally these days.
That kind of thing can be done with MS-DOS's 'EDIT.COM'. If you're nasty, it can be done with 'ed', The Standard UNIX Text Editor.
This is also where TEE's with attestation come in - https://redpill.ai/ for example (god I hate their name, but I am a fan of the product)
It’s been quite exciting that Qwen 3.8 Flash Next has come out: it really is similar to Opus 4.5, 4.6 for coding. Remarkably intelligent, and runs on a single DGX Spark, which I paid $4000 USD for
Sadly they’re now twice that price, which is a shame because I really want a second one!
I have a heretic modified version of it too, for when I want to use it for security and so on. Quite interesting
The 4090 I bought in '23 is selling for ~3x what I paid for it now in late '26. There's a huge squeeze going on with computer hardware using relatively cheap borrowed money. Relatively cheap money is flowing into a (artificially?) limited GPU/memory asset class. And this asset class is growing much faster in value than the interest on that money. So if you borrowed heavily to buy GPUs in the prior 3 years, you're likely coming out well ahead even if you are paying 8 to 10% on the money.
That dynamic will fuel further borrowing until we get to some kind of equilibrium, or some kind of washout occurs where interest rates spike higher and/or the value of GPUs and GPU services start leveling off or even declining. We might get a test soon as the FED has started hiking.
This is a great model on my Framework Strix Halo box as well. (also now about 2x what I paid for it in summer of '25)
Same here. I was planning to get a second, just got this one ten seconds before they became unobtanium. Here’s hoping Qwen 4 Flash is the same size!
Gotta find some rich friends in order to buy and host a H200 or two.
I’m excited for the refurb market in 7 years.
in 7 years you'll be soylent green as you're used to fuel the answer to 42
The circle of life in motion!
You really don't.
The cheapest one I see on ebay is $40,000... so yes, I'd say you need some pretty rich friends.
You can trust another party and do this by renting a few H200s. You cannot pool together with some friends without trusting another party.
You end up just weighing up the difference in trust between a vendor and a friend against the level of disinterest that they might have in your affairs.
A middle road is to use open-weight hosting providers, maybe non-US ones if you’re in the US.
Yes otherwise thoughts are crimes, effectively.
I agree up to a certain point, but there has to be some legal boundary between freedom of thought/speech and literally planning a crime. I'm protected under the First Amendment to say "someday I'll rob a bank" but not necessarily "I'll rob this bank on Friday and here's how I plan to do it".
I think you should be allowed to write that exact line in your journal. If you rob the bank that can be used as evidence against you, but in no way is it acceptable for private reflections alone to be used to arrest you. Or else every author who's written a novel with 'bad' characters would be arrestable.
I'm coming to saulpw's house tomorrow at noon with a gun, and I'm gonna make them pay.
If a policeman notices the sentence above on my phone screen during a routine traffic stop, the response you want him to take is... nothing?
For the record, under current US law, it is not illegal to have a sentence in your locally-stored notes on your phone outlining a plan to commit a crime. There has to be an overt act. The police in that instance could inform the intended victim, surveil you, etc, but they would not be able to successfully charge you with attempted murder. It's not illegal to be considering committing a crime, even if you have a tendency to write down your thoughts.
The law in this particular case, which seems to be intended for threats that you actually send to someone, is being interpreted broadly to apply to any "threat" that you transmit to a server. So in your hypothetical, the legality would depend on whether your notes are backed up to icloud or not.
>the legality would depend on whether your notes are backed up to icloud or not
I agree, and it's nuts.
This feels like less of an issue with anthropic per say as it is a broad reading/misuse of the law's original intent.
This is true for the general criminal conspiracy law, but be aware that an “overt act” doesn’t have to be an illegal act, just some action in furtherance of that crime. That can be purchasing a weapon, or scouting out a location. There may also be other laws in play depending on the specifics
Just having saulpw's physical address is already furtherance of that possible crime.
The commenters here are cute little HNers who think they have found a loophole in the law. They are not the first ones innover their head.
Spoiler: the law is written in words, and those words aren't strictly executed like in a computer program, they are interpreted by actual humans who can see what you are trying to do and will stamp it out.
Yup. HNers (and so many normies) believe that there are well defined rules they can follow and be safe. Spoiler, there is one law in the real world and that is might makes right, and you have no might, so therefore you will never be right. China for its many flaws at least is honest about this.
You're absolutely right, at least according to my own quick check on Gemini. I find this state of affairs amazing.
In my country, no "overt act" is required, but both here and in the US a "conspiracy to commit" charge requires an agreement with a second party. This is indeed consistent with a very broad interpretation of "no thought crimes".
Correct. Nevermind how ridiculously contrived the scenario you've just concocted is.
https://www.youtube.com/watch?v=14WE3A0PwVs
FYI this links to "Key & Peele - Rap Album Confessions"
its not contrived just because you recognize how silly the argument its arguing against is
> If a policeman notices the sentence above on my phone screen during a routine traffic stop, the response you want him to take is... nothing?
Anybody showing a cop their hacker news comments at a traffic stop should be arrested, for harassing the police
You joke, but part of the issue here is we're supposed to have laws that keep cops from reading over everything we type and using it to turn us all into crime suspects. There's a whole constitutional amendment about that, but our personal "papers and effects" that should be protected against unreasonable searches are increasingly really "owned" by others who are all too happy to snitch.
Part of me says that the solution is stop entering any personal data into any device and service you don't own, but I'm not sure if that's really what we want considering that there are zero private cell phones. Even desktops and laptops aren't 100% owned by you these days. The only thing you can really do is keep them offline 100% of the time so they can't spy on you, but that seems like a lot to demand.
Do you really think reading a snippet like that completely out of context should qualify as probable cause?
Yes!
When you read something describing in detail a person's intent to do something very bad, in a place where they write things that they intend to do, and which in the past they have in fact consistently done, you don't attach any significance to that at all?
I certainly think a police officer that stumbled across such a thing would be justified in asking follow-up questions like 'hey, what's this about you going to shoot someone?' and then maybe making an arrest based on your replies/demeanor.
'Probable cause' should involve a degree of certainty, because 'possible cause' would be altogether too loose of a standard. It's possible that you're intending to shootme and you just mentioned saulpw to throw other HN users off the scent. Possibilities are only limited by the assessor's imagination.
How can you prove that it's not just me writing a dark and morbid story?
How does the cop at the traffic stop know that your username is akoboldfrying? Did you tell them? If so, why, at a traffic stop, did you do that?
Are you posting threats on hacker news while you are driving? And the cop was close enough to see your username and what you wrote? Is that why you were pulled over?
Well in your example you've begun conspiring with a second party so that's not at all the same thing. You are at least free to plan all the crimes you'd like to arbitrary levels of detail in private. It's when you start acting things out (soliciting coconspirators, blackmailing targets, etc) that you cross the legal line.
The current situation is a weird one. Anthropic reported single party interactions (per the ToS and common sense), there's a statue about sending threats (as there clearly ought to be), then somehow the definition of the word "send" was tortured by the local police. If a crime has been committed here it's almost certainly an infraction by the local authority against the spirit of the law.
I totally agree. People seem to be stuck on the notion that we must not punish thought crimes, and have elevated this above all other considerations, when really it's just one among several.
However, those other respondents to your post seem to be accurately describing the current legal situation. I asked Gemini, and apparently "conspiring" to commit an offense requires an agreement with another person in both my country and the US, where an "overt act" is also required (that may not be incriminating by itself). I find this alarming. The fact that someone's private diary entry describing in detail a plot to kill me does not amount by itself to anything is... incredible to me.
The thing is plotting a crime is not illegal. Authors do it all the time. Usually against fictional characters, but some authors use a real world backdrop. And look over on StackOverflow. You can get some really weird ones that only make sense when you look into the details. When they manage to show up in the hot question list those details are missing. "How to kill a cat" comes to mind. (Note that you won't find it anymore--the question is AFIAK still there, but the title has been edited several times. The cat in question is the Unix command, not the feline. What do you do when you inadvertently tell the system to display the contents of a large binary file?) And, AFIAK still there, "How to kill Indiana Jones". (There are always the ancient mechanisms that somehow still work perfectly, springs and all. How to make something that actually would work?) And one I've heard of but not seen: "How to kill my wife", from some game I didn't recognize.
I’m sure the average UNIX admin has a trove of misinterpretable web searches.
fork a child and kill it - Google Search google.com/search?clie...067j0203j0i20i263j0i22
kill child and fork parent - Google S... google.com/search?clie...o...1...5.0j0171j35i39j
kill parent with fork - Google Search google.com/search?clie.....1... ..0171|35139|33116
kill parent without killing child - Goo... google.com/search?clie...4589.0j32j1.0....1......
kill child without killing grandchild -... google.com/search?clie...5.0j37. ....о...1...5..0j3
kill all children - Google Search google.com/search?q=ki...&hl=en-bg&client=safari
kill child with fork - Google Search cooale.com/search2o=ki &hl=en-ha& client=safari
You just said the words right here in this public vbenue, not even in private.
This argument holds no water at all.
Obviously, I'm not talking about the verbatim quotes I provided. There has to be some level of evidence that proves intent to commit a crime and the second quote is meant to represent that whole class of statements, but it depends on context. Any given quote won't constitute evidence in every case, but it will in the cases where it proves intent beyond a reasonable doubt.
I'm not sure why you think my argument holds no water when there are clear legal precedents that speech is not protected in some cases where there is "imminent lawless action".
https://en.wikipedia.org/wiki/Brandenburg_v._Ohio
I'm not a lawyer so take this with a grain of salt, but it seems like there is a big gap between speech that is "directed to inciting or producing imminent lawless action" and just saying that you will do something.
Depending on context saying "I'm going to rob the bank X tomorrow" might also count as a threat?
But, in either case, writing this in a private diary could not be incitement or a threat because you are not communicating with anybody except yourself.
action in the real world, i.e., not just words.
Lots of people today think thoughts are crimes
Or worse, misaligned with the Terms of Service.
This is not even an option in the UK. Communication felonies (dangerous speech, threatening behaviour) only demand potential audience.
It’s also an option to just not use chatbots
That's definitely something I'd consider if I had cash to spare for H200's! Unfortunately I think for most of us the price of self-hosting has to be 2-5x lower still.
Would be better if your friend is lawyer and take $1/yr(or higher) for the service so that attorney-client privilege can be applied.
edit: ah, future crime cannot be protected.
Surely a recurring automatically renewable $1 monthly/weekly/daily contract would solve the problem? Lawyer on a retainer can't really babble about your crimes and is required to protect the confidential information.
That's absolutely not how that works lol. There's a crime/fraud exception to the attorney-client privilege.
Lawyers have a duty of care to the court and they will absolutely drop your ass if you try to make them keep quiet about your crimes.
(This is not to be confused with them representing you in defense. You can tell them about your crimes if the government is trying to get you for said crimes.)
So IIUC, the correct nexus is to only hire a defense lawyer, and only discuss past crimes, probably in a rhetorical context. Otherwise attorney|client privilege only extends to non-criminal actions. Ok.
Way ahead of you bud, mac studio m5 ultra 256gb version is coming soon.... ordered 2x of em just in case. expensive as fuck but its a hedge against all this bullshit and more
Oh boy, I wish I had enough friends to afford H200 x 2
You don't need an H200 for that kind of use case.
A cheap second hand 10ish year old card like my radeon rx570 with 8GB of ram is plenty enough to run a small uncensored model with llama.cpp if all one wants is chitchatting with a clanker.
We are not talking about heavy coding use cases here.
Qwen3.8-27B is all you need.
Or you could just use notepad.
That's exactly what I do. I'm fortunate enough to use the latest preview version, which is known to us beta testers as Notepad With Microsoft Copilot™ and only uses up about 2% of my phone data for telemetry per day (Preview 1.0.912 actually used over 14% daily lol). Plus they got working set size down to 11 gigs and disk usage down to 36.3 gigs.
I'm on the other side of the political spectrum here AND YET I entirely wholeheartedly support what the person above wrote: use self-hosted LLMs, abliterated or otherwise. M
For many, AI chats are damn close to extensions of our minds: diaries. Those are supposed to be private.
Was what that woman wrote a credible threat? Was she blowing off steam? Without knowing her deeply, how can you tell?
Maintain your privacy. Prevent thought policing.
Preach! I am so tired of Anthropic’s safety team being the arbiters of what is right and wrong. If you so much as hint at impropriety you can have future sessions flagged ad infinitum when it comes to specific topics.
People say this all the time and it always makes me wonder what, exactly, were you doing?
I had a list of controllers that we forgot the usernames and passwords to. We knew it was some combination from about a half dozen of each, just not exactly what was what. A few hundred of them so a tedious task for a human to go through and validate them all.
Got flagged for attempting to have Codex write a quick script to basically dictionary attack my own infrastructure with a tiny dictionary file.
A bit silly, but it doesn’t take a whole lot.
Lead time is like a year tho no?
I agree, but someone will say fake child porn.
I'm not saying fake child porn should be allowed or not-allowed, just showing there exist possible exceptions and rationalizations for them even without two parties.
Fake CP is a victimless crime. It is debatable if victimless "crimes" (like being gay in your own home, which is illegal in way too many countries; or eutanasia; or eating pineapple pizza) should be punished at all.
I understand moral panic, disgust, etc, but rationally speaking.
There is an argument that there is a victim, which is all children in that persons vicinity, and all people who care about them, and even society itself as a whole. And the harm is the reasonable elevated risk of harm, combined with the particular helplessness of the targets who are incapable and not responsible for looking out for themselves.
Again I'm not saying it is or isn't a valid argiment. There are 50 easy counter arguments without even trying, but that doesn't mean there is no argument to be had there and there might be 50 counter-counter-arguments in the end.
This is akin to calling all men rapists only because they theoretically can. "All men until no man".
And even in this case, this has nothing to do with fake CP. Surely the threat exists due to the location of a potential perpetrator, regardless if they have fake CP, real CP or no CP at all.
> Florida Statute 836.10 makes it a second-degree felony to send, post, or transmit a written or electronic record threatening to kill or injure someone [...] The communication must be made in a manner in which another person may view it.
Which it clearly wasn't, right? I mean, okay, in this case, the message did get reviewed by another person, but that's obviously an exceptional circumstance.
If I write something down on a piece of paper, and someone else goes through my garbage and finds it, is my note "communication made in a manner in which another person may view it"? It was clearly intended to be a private note!
It is worth mentioning that state law may not abridge people's rights which are protected under the Constitution. It is perfectly legal under US law to make any number of violent statements as long as they do not rise to the level of true threats.
https://icap.law.georgetown.edu/wp-content/uploads/2026/02/T...
I am not a lawyer, but I find it hard to believe this statement meets that bar.
Given that there's not really an objective measure of the trueness of a threat, this seems like the sort of thing that a court decides the merit of after charges, though.
Somewhat relatedly, I used to do tech support for a very large e-discovery vendor. The number of companies that configure their systems to archive (save, index and make discoverable) the contents of the /drafts folder in email clients is astounding. Surveillance of composition, rather than surveillance of communication, seems wrong.
The reason for doing that is to get all of the email in a system, as required by whatever rules apply.
Today I learned about "foldering"[1], which uses the drafts folder of email systems as a "dead drop" between multiple participants. It goes back to at least 2005.
Back when I was in IT, circa 2010, I learned that several of the users of our systems routinely relied on the "deleted items" folder of Microsoft Outlook as a filing system, with multiple gigabytes important files stored there. 8(
Our customs are routinely ignored by everyone else.
[1] https://en.wikipedia.org/wiki/Foldering
> Today I learned about "foldering"[1], which uses the drafts folder of email systems as a "dead drop" between multiple participants. It goes back to at least 2005.
Former head of the CIA, David Petreaus, was using the drafts folder of a shared Gmail account to communicate with his mistress.
I guess that’s easier than Dropbox
And, critically, not blocked by bluecoat, whereas Dropbox is.
The reason so many people used the deleted items folder as a filing system is exchange didn't set a quota on that folder and by default would only get emptied by a manual emptying. So back in the days when a user could have a 100mb mailbox quota 'unlimited' space could start looking very attractive. Especially if that quota was 5 years old because the server hadn't grown with the times, since what business wants to spend money on a 'cost center' that won't improve profits.
Then these people didn't change their ways when they were given adequate space.
It was used by the subject of this book as well, which I think predates 2005.
The Spy's Son: The True Story of the Highest-Ranking CIA Officer Ever Convicted of Espionage and the Son He Trained to Spy for Russia
https://www.amazon.com/dp/0802125190
> If I write something down on a piece of paper
At some point I think these sorts of analogies break down, as the setup becomes too foreign to what we're more concretely used to.
In this case: there is no pen or paper which can store what you write on a replicated set of servers across the world, with an accompanying ToS telling you how that will be treated/used.
Although I have little sympathy for this women (both her intent and stupidity), I do agree this is a dangerous thing.
But that's because you are a technologist who understands how servers work and has to think about where data lives. Meanwhile, mainstream software design increasingly blurs the line between local and remote and makes it hard to tell what is what. How can she be expected to know?
Ai labs are 80% surveillance machines. I will never understand how people trust these services with anything personal, emotional, medical, financial, whatever.
I don't know why people are even discussing this case like these companies will do anything on behalf of a consumer or think ethically at all.
Bots mass read and file and report all prompts that get categorized a certain way. They store everything else regardless. This will never change. It will only be exploited more and more. That's how this type of technology is deployed and progresses. Look at any other parallel. Like cameras or microphones.
I gave up trying to avoid to be surveilled. It takes a lot of energy, I lose out on too much utility, I make it into the data anyway transitively from sources I don't control, and finally I don't feel any more interesting than the rest of the world. If I get uniquely screwed over somehow, so be it.
There's a famous quote about that...
I don't think it's worth dedicating even 2% of someone's day to avoiding surveillance. But I do think it's sad how many people don't realize all the utility they are gaining is lost once surveilled. There are ways around majorly bad surveillance activities that don't cost much money or time. The 80-20 is completely worth it.
Are you saying AI might become the Big Brother?
It's the only thing capable of being that. So yes.
I assume you are kidding. But yea and it's not might, it's the point.
Anthropic itself is legally a person, whereas your bin is not.
>another person may view it
LLMs seem to be moving toward personhood.
And after all the LLM learns from user posts too, and if everybody starts posting their darkest desires, fantasies, plans it may skew the "alignment" to say the least. Lets hope that the AGI level doesn't necessarily come with cheating, lying, religious fervor, power lust or whatever other sideeffects have been observed in human intelligence.
Both OpenAI and Anthropic state in their terms and policies that they can access, retain, and review user inputs and conversations (yes by either an LLM or a human). So I guess at this point it is now up to a judge (or a jury) to determine if the person is dangerous.
The logical next step would be, to create psychological profiles of each user and relate those to authorities.
Selling analogous profiles of companies using their services to the highest bidder would be another idea.
Whether intentionally or not, they market their product as suitable for all kinds of things that it clearly isn't when input is being used that way.
That's the previously logical step - that's already happening, which is why we have this story.
Hopefully we at least get some cool Tom Cruise action scenes when we create our own shitty version of Minority Report
If the threat was to a private citizen and not the sheriffs office, I don’t believe charges would have ever been brought
> "may view it"
If we assume that "may" here means "could somehow" and not "is authorized to", the legal action after revelation seems correct as written (ignoring whether the person viewing it has any relation to the person being threatened) unless the law gets struck down as unconstitutional. The question is only whether Anthropic should or should not report it.
> "If I write ... It was clearly intended to be a private note!"
The law as written doesn't appear to distinguish about intent of privacy. Also, if you're in the habit of writing notes to yourself, I guess maybe don't write down the one that says you're going to shoot up the sheriff's office.
How can you charge someone for making a threat when you only read the threat by spying on them? Surely that has to be thrown out in court? They didn’t actually send the threat to anyone, you just obtained it by spying.
> when you only read the threat by spying on them
The AI companies have clauses in their user agreements saying they can review content flagged as harmful. It’s not legally spying.
If you recall previous outrage about ChatGPT being used in cases of suicides or shootings, this is the result. Every time a crime was committed and the police found ChatGPT history about the crime, the media turned it into a frenzy. So the AI labs added safety filters to their consumer plans that detect threats of violence, escalate them to human review, and report to the police.
Spying is not the right analogy because the information was given to the police by a third party which had a EULA saying they would do this. A more analogous situation would be someone reading another person’s diary and then turning it into the police department. There might be some limitation in the law that makes the evidence inadmissible because it was not intended to be shared with anyone, but that’s a separate decision.
Not only that they can review flagged content, but they tend to have separate retention policies for flagged content. Anthropic's is this: "We retain inputs and outputs for up to 2 years and trust and safety classification scores for up to 7 years if your chat or session is flagged by our automated trust and safety systems as violating our Usage Policy."
So don't run for office or anything like that. Someone, somewhere will have a contact that will get that.
It kinda is, actually. If the LLM had responded with 'woah, are you serious? That sounds like a crime and I can't just ignore that, it's made clear to the customer that such statements are out of bounds even if they were meant hyperbolically or humorously. But if someone crosses the guardrails and the system silently reports them, that's very much spying.
Obviously, it's hard to judge exactly what was appropriate there because we're being asked to extrapolate from a two word quote about the customer intending to "shoot up" the sheriff's office. Consider the following two statements, which express quite different levels of intentionality.
I got a $200 ticket from a sheriff's deputy today for throwing away an apple core. I'm so mad. I'd like to shoot up their office!
Those sheriff's deputies have exhausted my last reservoir of patience. I'm going to shoot up the department. They'll be sorry when they're sprawled all over the floor bleeding out from saucer-sized shotgun slug wounds. I can't wait to hear the screaming and crying of their miserable families!!"
I'm guessing that the diary entry was a more casual expression similar to the first statement, or they police would have quoted more of the statement to emphasize the apparent severity of the risk but it's hard to say without reading the charging documents.
yeah why wouldn't the llm push back? I said "fuck you" to gemini once and it replied to watch my manners, and when I realized that it could delete my emails no problem I'm now all please and thank you, problem solved.
>Spying is not the right analogy because the information was given to the police by a third party which had a EULA saying they would do this. A more analogous situation would be someone reading another person’s diary and then turning it into the police department.
This is spying with extra steps couched in corporate speak.
I was responding to a question about the legal case. The police did not perform any spying.
Frustrations about Anthropic’s EULA are a separate matter.
Was it claimed that the police did any spying?
Presumably, Anthropic did the spying and the reporting.
You argued that it is not spying, since the spying may have been made sufficiently explicit in the ToS/EULA.
This raises the question: Does announcing a spying operation mean that it is no longer spying? I've never heard that perspective before.
> Does announcing a spying operation mean that it is no longer spying?
Well, kind of, yeah; the dictionary definition of spying requires secrecy and lack of consent.
> to secretly collect and report information about the activities of another country or organization[0]
The only real debate is whether or not having a clause tucked away in a EULA that few people read makes it a secret. If Anthropic had a big flashing red banner that said "FYI we automatically flag and review any conversations about illegal things!!" on the front page nobody would call it spying.
[0] https://dictionary.cambridge.org/dictionary/english/spying
I would call it spying in this sense at a minimum if individual people don't know whether their conversations were stored or disclosed in a way they don't want. For example, suppose someone said "we will monitor the activities of 10% of people". You don't know if you're in that 10% or not, but I would still want to call that spying.
A less central case would be when you clearly do know about the activity but you can't quite see the details, like with behavioral ad targeting or something. It feels pretty normal to me to call that spying even if it's disclosed to everyone and certainly happens to everyone, but it's also a less central example of the concept.
Anthropic could put a big flashing warning text at the top of every chat that says “We are spying on you and will report anything scary to the police!” and it would not make any difference in this case.
You can call it anything you like, but only the legal definitions matter for the legal case.
After working on several court cases about surveillance activities, I'm definitely aware that whether I call something spying or not has little relationship to whether courts will think it's legal.
> spying requires secrecy...
Eh. Both Superpowers knew that they were spying on each other all the time, and that was still considered to be spying. But feel free to replace the word "spying" with the phrase "clandestine largely-automated mass surveillance" if it makes you more comfortable.
> ...and lack of consent.
Given
* the fact that the contracts one is required to "agree" to in order to use most services are often novella-length or longer, and frequently include by reference other contracts of similar length
* that nearly all contracts like this have a clause where not only does the powerful party reserve the -very frequently-exercised- right to change the terms of the contract without any prior notice, but said party presumes that you automatically accept the rewritten contract and gives you no option to negotiate
I'd argue that the real situation on the ground -in the US, at least- is that "consumers" have consented to approximately zero of the contracts that -despite that lack of consent- legally bind them.
> If Anthropic had a big flashing red banner that said "FYI we automatically flag and review any conversations about illegal things!!" on the front page nobody would call it spying.
If you change the situation then yes you can in fact change our responses. The problem is you then are no longer talking about the original situation.
It also bears mentioning that providing a dictionary link to “spying” is pretty patronizing/passive aggressive. On par with sending a basic Wikipedia page. You didn’t even bother to post the definition you want to apply.
No one claimed any case would be "thrown out for spying." The legal definition of spying is also not particularly relevant to the argument in the initial comment.
The initial comment instead questioned how someone could be accused of making a threat if they did not realize anyone would read their private content. You probably also can not insult someone with a statement you never expected anyone but you will ever read.
I didn’t say they did
Yes, you are right. My reply was written for a different comment on the same level. Sorry for the confusion!
All good
> Presumably, Anthropic did the spying and the reporting.
You don’t need to presume. Anthropic reported it.
“Spying” as a legal concept has a definition that does not apply here. You could say they were “spying” in the sense that they read someone’s input, but that’s literally what they said they were going to do in the agreement when the person signed up.
So I responded to the question about the case being thrown out for “spying” by trying to show that the word doesn’t apply in the legal sense. If you sign up for a service that says “Hey we’re going to monitor your chats and might report things to the authorities” and then they monitor your chats and report things to the authorities, you should not expect the case to be thrown out for “spying”.
There is no spying by any definition. It's a chatbot, not a diary. Anthropic is expected to read the message and respond to it in some way. If you sent an email to a colleague threatening violence, you would not be surprised to find out it was reported.
> Anthropic is expected to read the message
"Anthropic" does not read messages, it's an abstract entity involving many humans and computers, so let's be specific wherever possible.
> and respond to it in some way.
The computer is supposed to respond in a specific way that doesn't involve humans. Any reading/actions by humans is entirely separate and not expected.
> If you sent an email to a colleague threatening violence, you would not be surprised to find out it was reported.
And if I didn't send it, I would be surprised.
Yeah it’s like someone going through your email drafts. I keep seeing people using examples that are clearly not analogous. If I send something to a person or say it out loud knowingly to a person it changes the situation entirely. I am aware of others, I am aware I sent it to be received and read. That is clearly not what happened here.
Extra steps couched in corporate speak is often the defining line that defines whether something is technically legal or not.
I’m speaking from a functional/ethical framework to be clear. I’m just expressing frustration, not challenging the comment. Could’ve been clearer on my end there.
Well, let's say that you have a regular customer at a bar.
They get friendly and loose-lipped with the bartender over the span of months. Eventually they let slip that they plan on killing their spouse for a life insurance payout. At first the bartender thinks they're joking, but it becomes evident that there's an actual plan being acted upon and someone's life is very likely in imminent danger.
Does the bartender have a responsibility to go to the police?
Depends on the country. In some places, there is no legal repercussions for not reporting this to the police; in some, it is an actual crime in itself.
In this case, let's assume the country is the United States, and the state is... oh, of course it is... the state is Florida.
In this case you inform the police only after it has happened, so the civilized world can get rid of the two Floridans at the same time.
/s obviously
For better or for worse, Florida (wo)Man is eternal.
Not the same situation. It was unintended but they are the ones who spoke out loud to a person they knew was present and in a public venue.
It's not that different.
It's not 2005 anymore. If you think that there isn't any way for the people operating an online service to surface your activity on that service, or that there is but those people aren't doing so, there might not be anything left to convince you of it.
What you put into a text box online can be used against you. Period. You have to act accordingly.
What? Why on earth would you think I don’t understand that? What about my response says otherwise?
If you are a business and I am using your services, it is pretty damn unethical and wrong to vacuum up my data and hand it off. Yes I know they all do it, I’m not naive. But a lot of comparisons people are making are not analogous to straight up sending a message or saying something to someone directly. Companies thrive on opacity and convoluted EULA’s to spy on us without clearly saying they are. Please do not talk down to me just because I’m talking about how things should be, about what is right and wrong, rather than blindly going “well AkShOoAlY you signed the thing and you should know that everyone is always trying to screw you so just live accordingly.“ I’m not OK with that, that is not how I want to live my life even if I am forced to, and I am going to make it known that I take issue with it.
Corporate surveillance is a blight, it is literally harming our society. Every time you tell people essentially “deal with it” you are reinforcing the current situation. Expect better from companies and society as a whole. Demand better.
You are clearly a smart person and you want to have a discussion, so is this the argument you want to make? Effectively defending companies by telling people to just suffer their abuses or keep their heads down?
> This is spying with extra steps couched in corporate speak.
Calling something names doesn't invalidate it. It only invalidates what point you're trying to make.
“Calling something names”? I described what occurred. I didn’t (and can’t) personally insult the concept of EULA’s, as much as I’d love to.
Many clankers deny data retention or spying on the user if you ask them. That should be completely illegal.
Then, you can write anything in an EULA but it is not automatically legal either.
This seems to be the statute: https://www.leg.state.fl.us/Statutes/index.cfm?App_mode=Disp...
With the obvious IANAL, it doesn't seem to rely on the message be sent to the person being threatened. The specific segment is "in any manner in which it may be viewed by another person".
This may be one of those cases where we get to find out how courts view SaaS platforms.
Interesting that it exempts telephone calls. Why don't we treat other messaging services like phone calls?
It looks like a prior statue covered that: https://www.leg.state.fl.us/Statutes/index.cfm?App_mode=Disp...
The subjective element of crime (i.e. doing it on purpose) is fundamental also in the US legal system. If the person wasn't aware that someone else might see their messages, it should be hard to claim that they committed the crime.
According to Gemini, "Florida appellate courts have overturned juvenile convictions [based on this law] when the state could not prove the person subjectively intended for the record to be seen."
The prosecution will ruin her life regardless of the outcome.
Exactly, can you threaten someone without them receiving the threat? If this is not thrown out, Minority Report will actually happen.
We are snowballing to Minority Report ...
If the AI recommends murder and you exhibit a pattern of following AI advice are you guilty of precrime
this is almost certainly what anthropic is hoping for here - a judgement that says there is no point in them continuing to monitor and report this behaviour
> Surely that has to be thrown out in court?
The prosecutors likely know this and expect it. But there's enough gray area here for them to make the argument, and it's hard to prove malicious prosecution, so they know they'll get away with it. It's just about sending a message to the public - they don't care whether a conviction sticks. Just politics.
The prosecutors aren't on the hook, anyway. They have absolute immunity. The decision to charge is protected. The prosecutor would have to have done one of the few, enumerable things outside the scope of the role, like conducting an investigation without probable cause or hiding exculpatory evidence.
the argument to be made is that allowing anthropic to see it constitutes sending the threat.
sandbox your ai.
That is not what sandboxing solves. A good sandbox would inject credentials into provider API calls so that the model never sees credentials, but the provider is still going to see the transcript. Sandboxes do not require or imply that there is a local model. Sandboxes limit what the agent can access on the host machine as well as the network and public internet.
>Sandboxes limit what the agent can access on the host machine as well as the network and public internet.
this is exactly what I meant. I am presuming the danger is AI reacting to personal notes that it reads on your computer, like a diary, and you should not allow the tools to have access to those documents.
It was sent in a chat message. If you sandbox your coding harness, and then go type threats into it, you've sent it to anthropic.
ah, okay - yeah kind of a 'shocked pikachu face' then. So maybe more of a PSA is needed that what you say to the model and your uploaded files is akin to speaking all the content out loud to the company hosting the model.
I dont really like this direction but it is what it is right now
how does sandbox help in this case when you use a provider like anthropic/openai?
If you're still using a provider like this then you didn't sandbox the AI. You still need to follow the instruction.
Another way to interpret this is that they are legally presuming that you already have sandboxed their product and anything it sees or has access to is intentional.
Any failure to understand what it can access or what it has permission to see from the user's end is presumably not their problem. Regardless of what the user specifically asks of the tool.
It's not quite spying when you willingly hand over this information and agree to terms of service. You're data is not considered yours alone.
It still shocks me the number of people I know who freely let agents on devices that contain unencrypted private keys, freely dump internal data into cloud models and generally don't give a second thought about any of it being trained on, inevitably leaked one day in a db breach or read by providers. I find it's best to consider any data put into a cloud model the same as if it were posted publicly online, since that is the very possible eventual end result.
Hopefully more of these stories push people towards local models :)
It's legal to spy if the terms of service say so
Cops will charge them to let the courts decide
I was thinking the same. If the evidence was not obtained with a proper court order wouldn’t this result in a mistrial?
If you overhear someone, in the privacy of their house, threatening to murder someone and go to the police, surely you don't expect this report being thrown out and you being yourself charged with the violation of someone's privacy instead?
IIRC if the evidence wasn't lawfully gathered (which it sounds like it was, tbh) then it wouldn't be a mistrial, it would be thrown out and then the prosecution wouldn't have any evidence of any crime.
what if it's just testing the AI to see how it responds
A threat sent by mail is still a sent threat even if nobody ever opens the envelope to read it. The crime is in the sending. This woman used an online resources, one which involves transmitting everything across innumerable state lines. I am surprised she isn't up on federal charges.
Note that the law doesn't forbid the writing of a threat. You have to send it to someone. Had she kept it in a book under her bed, she would not be in trouble. But she sent it to a website/service/LLM portal.
>> It is unlawful for any person to send, post, or transmit, or procure the sending, posting, or transmission of, a writing or other record, including an electronic record, in any manner in which it may be viewed by another person
If you draft an email threatening someone and delete it without sending have you committed a felony because someone at Google could be reading your drafts box, stored in a datacenter across state lines?
Honestly, I'm equally fascinated by the way email has changed. 30 years ago when you drafted an email but didn't send it, it was only on your local machine. There was no SMTP. 20 years ago, it might be a 50/50 shot as to whether you "transmitted" it to your "Drafts" folder if you were using IMAP instead of POP3 to read it.
We really need a way to make it clear to users when, through the normal operation of software, they are "sending" data to a third party (usually the software developer) and when they are not. This is definitely not clear/knowable to regular users, and it's kind of hard to figure out even if you're a computer expert. Even software that "runs locally" now sends innumerable amounts of stuff back to the developer, and they don't always disclose it.
This is a huge privacy problem that is only going to get worse.
Sounds reasonable. Google's bots could pick that up easily and forward if for human review.
FYI, the use of drafts folders to transmit messages has been used by terrorists. This is likely where CIA director David Petraeus got the idea when he needed a secure way to chat with his mistress.
https://www.findlaw.com/legalblogs/technologist/gen-petraeus...
How about you save a text file on your Desktop or in your Documents folder, which your computer has bullied you into syncing to OneDrive or iCloud. Fair game to get sent to jail for that?
Ah, a simpler and more innocent time of government scandals. I miss it. Now the messages are on White House stationery and they declare themselves above the law.
Be happy for where we are in 10 years we'll living in a world of llm based decisions where all conversations, actions, thoughts will be monitored. Where llms can excuse any decision. At least we have an all show government with at best partial results who generally retreats.
It sounds extremely unreasonable to me for "bots could pick it up" to transmute a private note into a felony threat.
It "sounds reasonable" that the exact same action could be a crime or not, depending on how an engineer implemented a feature?
What if she mailed it to herself?
What if she put it in a locked box before shipping it to herself UPS, and she has the only key?
What if instead of UPS, she hired a moving company to move the locked box?
What if she wrote it electronically in diary.txt, but it was backed up to a cloud provider?
--
I'm guessing there's some sort of "reasonable expectation of privacy" for certain activities. We're going to find out what Florida courts think about this new medium.
We'll only find out what the courts think when this happens to someone with a lot of money. It takes a real legal fight to push it high enough to become precedence. She'll be pushed to plea out.
Surely this is the wrong side of what "sending" here will be interpreted as?
Saving is not sending ie passive vs active act.
Everything you "save" on an online service gets "sent" to someone, be that a person or a computer, more often than not across state lines.
People really need to stop parroting this "across state lines" thing.
> in any manner in which it may be viewed by another person
Does the person have to know (or at least believe) that it will be viewed by another person?
She likely didn't think anyone would view it. Honestly, even as a career software developer I don't think it is unreasonable to think know would would see what she wrote to an AI. I assume most of what I write to an AI is not viewed by any other human, based simply on the quantity of messages sent back and forth to AIs, I would assume a vast majority are not read by another human.
What if she had written this into google docs, and she kept a diary there? That also crosses state lines, and is transmitted to another location.
I get were you are coming from but this all feels like it needs more context to make a better judgement.
You can argue from technicalities but they would need to prove intent.
> Florida Statute 836.10 makes it a second-degree felony to send, post, or transmit a written or electronic record threatening to kill or injure someone, carry out a mass shooting, or commit an act of terrorism. The communication must be made in a manner in which another person may view it.
I think Anthropic did the right thing here; but the sheriff's office are probably demonstrating why she dislikes them. Writing a diary entry to a chatbot is clearly not how this law was intended to be used.
EDIT: Actually, on reflection, making this report to the people she was upset about was probably not the right call. If they'd sent it to the FBI, there'd be a much lower chance that someone felt the need to assert their "authority".
>> I think Anthropic did the right thing here
This is the paradoxical times we live in right now.
Don't do something? She walks into the office and start shooting the place up. Several officers and innocent people are killed. Cue the media claiming, "You should've known she was talking about this an AI bot! Why didn't the bot tell anybody she was planning a mass shooting?!"
Do something? She gets rolled up by the cops and questioned about what she was talking about and brought to the cop station and interviewed. Cue the media claiming, "This is an unethical way to use AI, this is an infringement on free speech! This is authoritarian!"
I believe in free speech as much as the next person. But in this day and age, its almost better to be safe than to have to explain to someone's loved ones you had to chance to prevent this and did nothing.
> This is the paradoxical times we live in right now.
It's always been complicated like this. That's why certain professions (psych, lawyer, clergy) come with rules around when and if disclosure is allowed[ required, and/or admissible].
> This is the paradoxical times we live in right now.
This quote is pretty old:
> Those who would give up essential liberty, to purchase a little temporary safety, deserve neither liberty nor safety.
> its almost better to be safe than to have to explain to someone's loved ones you had to chance to prevent this and did nothing.
An authoritarian government isn't safe. That's why safety is also not deserved when you go chasing for a little of it at the cost of essential freedoms.
First and foremost journalism is the business of turning news into attention (and ad revenue), not informing the public.
s/journalism/poor journalism/tnx
Mmm.
As a Brit, I'm aware of https://en.wikipedia.org/wiki/Twitter_joke_trial
I can't say I actually disagree with the initial prosecution. The penalty was a fine, likely less than the cost of investigating it.
Intended as a joke? Blowing off steam? I can understand that, but given the number of people on social media is large enough to include genuinely unhinged people, you can't expect anyone who receives such as message to take them as a joke.
Same with AI use. A billion users, you have to assume some of them are actually sincere if they write about any act of violence, from self-harm to a plan to steal a nuke and use it in a false-flag attack to trigger WW3 and everything between.
> you can't expect anyone who receives such as message to take them as a joke
That's a distinction that matters to me. Sending a spicy note to an LLM isn't remotely the same thing as posting it on social media where the entire world can read it.
I kinda agree, but on the other hand anything you send to an LLM has to be viewed through the lens that only an automated system (i.e. an LLM) is capable of even handling such a tsunami of natural language.
It absolutely will misclassify things, it doesn't know any better.
(We should all wish each other good luck, because we're going to need it).
They’d be in an easier position if they built the system such that it was impossible for them to know what people are writing. They might catch a little flack from people who want them to surveil all their customers, but by and large people seem to accept “we take technological measures to ensure privacy and that means we can’t spot crimes.”
So this conundrum is at least partly of their own making.
> They’d be in an easier position if they built the system such that it was impossible for them to know what people are writing. They might catch a little flack from people who want them to surveil all their customers
The companies making these AI chat bots are the people who most want to surveil all their customers. The plan is (or will end up being): spy on what their customers are saying to their chatbots, use the conversations to further train/improve the AI and to increase the user's engagement with it, mine the conversations for every scrap of private/personal data and build dossiers on the customers, let other companies and governments pay them for that data or at least turn the chatbot into a shill/manipulator targeting customers based on the contents of their dossier, then use the contents of their customers dossiers for anything else they want.
If they actually set up their services to be private it would kill almost all of the expected value and also cut them off from their best remaining source of training data that isn't the AI generated slop their products are increasingly polluting the internet and the rest of media with.
This is the only way to recoope the spending. Soon after an AI tax and every cititzen gets a free llm account at some provider whether they like it or not.
It feels icky, and my default is not to side with megacorps engaged in blanket surveillance, but I can’t really fault Anthropic here. The correct setup should be a law that protects this sort of interaction with a chat bot as privileged, along the lines of HIPAA-mode. Use a classifier or some sort of “private mode” toggle to tell the platform you’re engaging in privileged communication ala dear diary, and then a much higher legal standard needs to apply to protecting that data (no training, same protections as doctor / psychologist interactions). Even a therapist has a legal duty-to-report in certain situations.
I’m going to guess that planning a mass shooting clears any hurdle that would be in place here.
While the privacy around ai chatbots is rotten in general, I can’t fault anyone who reported this.
Maybe not one that reported it, but the ‘we scan every message you sent into this for disallowed speech’ part is what gives me pause.
They shouldn’t be in a position to report it at all.
Yes in general I don't like that everything you do on a computer is sent to some 3rd party to scan. But the moment they did receive this message it's clear the only right move is to act on it.
Is it really "blanket surveillance" when it only sees exactly what you put into it? That's like saying you're being filmed against your will while... filming yourself.
I get where you’re coming from, but I think if the progression of technology has shown us anything, it’s that there will be constant competitive and social pressures being applied to use these tools more and more and that pressure will race far ahead of any privacy, consumer protections, public education and societal wellbeing counter forces. So opt-out ends up being a rapidly shrinking iceberg in practice.
"slippery slope" is an old phrase for a reason.
Counterpoint, if you give the agent access to your files - which happens to include the Notepad diary you've been writing since your teens - and it makes a similar conclusion about something you privately wrote, would have the same opinion?
Yes, and it's weird that on a form of technologists we're going with the idea that all of these systems we use every day to hold our personal private information that we are constantly ensured is secured against anyone unauthorized from accessing it is actually snitching to the cops just because someone else is hosting it on our behalf.
The 3rd Party Doctrine destroyed the 4th amendment and is the reason privacy respecting software has to play legal games. E2EE while a good security practice shouldn't be necessary to protect you against the cops rummaging around your stuff. The bar to establish that information is private shouldn't be "literally mathematically inaccessible but the cops are still allowed to try."
Would you use an AI service knowing that they are inclined to turn you in to police if they detect illegal activity?
In a "three felonies a day" universe?
> Would you use an AI service knowing that they are inclined to turn you in to police if they detect illegal activity?
The appropriate question is whether I want to live in Florida. This is much more a Florida law problem than an AI company problem.
It would apply if you happened to use Office 365 to write your diary.
Sadly, this isn't a Florida problem.
I don’t live in the US. But this kind of broad law is hard to implement without surveilling all users, and it has multiple side effects.
What happens if I use Claude or ChatGPT to research sensitive social topics? Would that be considered a social network interaction and used against me when I apply for a visa?
Many governments (especially in Latin America) copy what the US does, meaning that similar laws will be pushed sooner or later.
> What happens if I use Claude or ChatGPT to research sensitive social topics?
Anthropic/OpenAI is not obligated to report your use, as long as it's not violating some terribly written law (like the Florida law). The government won't know about it, so no, you will not be denied a visa.
> Many governments (especially in Latin America) copy what the US does, meaning that similar laws will be pushed sooner or later.
I honestly don't know, but I suspect most US states don't have such an overreaching law.
Legally they have to.
Would you use a lawyer knowing they are inclined to turn you into the police if you talk about committing a crime in the future?
The law is the law.
We should be happy about this as for once the AI companies did the right thing.
This is slippery. Many people do all of their journaling inside of Google or Apple cloud products. Some even write up their intentions to do bad things.
Does all writing now have to be scanned for thought crime?
That's an interesting wrinkle that is rather tough to work through.
To me, journaling your intent in a private journal, whether that's an Apple Journal/Note or a Moleskine in your drawer, feels qualitatively different in some way. But I'm not sure why.
So "I'm going to shoot up the police station" written in your own journal feels somewhat different than "I'm going to shoot up the police station" said to a system that might be able to _interpret_ or _act_ on what was said in some way. Did I just give AI a legal duty, or even a soul I didn't think it had before? I've written up about three or four "what about this, what about that" and deleted them all.
Your LG television transcribes every word you say in the living room. How would you feel if they began collaborating with police? Have you said any sentence that would seem incriminating out of context?
"If you give me six lines written by the hand of the most honest of men, I will find something in them which will hang him."
Yes.
But that service is a process on MY computer, loaded into RAM, with an abliterated model.
I don't trust 3rd party networks from abusing any data I give them.
You omitted the part in the diary entry about shooting up sheriff’s office
You're missing the part where it's a diary entry, so the actual content is irrelevant. Her only mistake was not realizing that her diary wasn't private.
Because it's not relevant. They're not charging her with conspiracy, which is what they would do if she'd actually done anything concrete towards making that happen. She didn't email it, or text it, or post it on Facebook or Twitter or Discord or a message board, which is what this law is clearly about.
I have told llms all kinds of stories to find out what its answers would be. I always make it sound like it is the truth to make sure the AI answers in a way that it would if somebody actually said this. I also tested internal flagging systems of the ai company I work at with the most evil things a person can ever say to find out if it would flag them.
Of course I did not mean any of that stuff, but how can you make sure a human reviewer knows you did not mean it while the llm does not know that you did not mean it.
I guess its a miracle I am not in jail yet.
Flagging people for anything said to an llm sounds wrong to me because an LLM is not a real person and while some people put in their internal thoughts, others just roleplay and the two are inseparable just from reading it.
Don’t worry, they’ll store those messages forever and incarcerate you at their convenience.
I was sure I couldn't be the only one curious to push LLMs to their limits. Though these days it's much tougher, mostly impossible to get them to react in unforeseen ways to horrendous scenarios.
This is exactly the typical use I make of the llm.
Adding: - I typically ask questions in the I form, regardless for whom or why I ask for. - Gemini chats quite often end when it starts recommending psychological council or a suicide line, to talk about my problems. It apparently detects a persistent tendency to not agree with the party line. So it makes sense I must be suicidal ;-
But sure, as llm's start to babysit us, and know our inner dialog better than anyone else, we'll soon be debugging their opinion/behavior/co-existence/authority, when it comes to reporting people to the authorities, or taking on tasks in society in general. We'll hire doctors to cure our psychological profile from our record (Total Recall).
A Minority Report like this shouldn't cause a referral to the police.
I don't understand how she violated this law:
> Heller faces a charge of making a written threat of violence under Florida law. Florida Statute 836.10 makes it a second-degree felony to send, post, or transmit a written or electronic record threatening to kill or injure someone, carry out a mass shooting, or commit an act of terrorism.
She didn't threaten anything, she wrote down that she was going to do it. A "threat" is more than a mere statement, especially when written in what is described as a "diary".
> A Florida woman is facing felony charges after she used Claude as a diary and allegedly wrote that she planned to "shoot up" the Sheriff's office.
Obviously I don't want anyone to shoot up anything, but this seems like a weak case legally speaking.
I think it's fair to say this is a gray area. Clearly it was transmitted.
I can certainly threaten you harm and send it to not-you and you're still clearly in danger even if it wasnt transmitted to you. So the question becomes did she transmit it to someone? Clearly yes she transmitted it to Anthropic. But she clearly intended to send it to Claude, an inanimate object.
Claude's terms of service makes it very clear that their employees will read messages[0] to determine that they don't contain the things that these messages contained[1].
> Review is needed to enforce our Usage Policy... designated members of our Trust & Safety team may access this data on a need-to-know basis as a part of their evaluation process.
[0]: https://privacy.claude.com/en/articles/10458704-how-does-ant...
[1]: https://www.anthropic.com/legal/aup
I highly doubt she read the terms.
The critical part of a "threat" is that the perpetrator takes some intentional method to deliver it.
I doubt it too, but they're legally binding, and extremely permissive in favor of the company far past the point of technical necessity...
...But since they exist, the company acted more responsibly than if they had simply ignored the data they chose to observe. Which is both the bare minimum given the circumstances, but also strangely absent almost all the time in the industry (including within Anthropic).
I am not a lawyer, but I have the feeling that reporting a discovered terrorist manifesto or something similar is in the spirit of the law.
Nobody is saying that Anthropic didn't have the right to read it. They obviously do.
I am suggesting that the criminal case against her lacks mens rea because inside of her own mind she did not expect that anyone would read it.
A terrorist manifesto is in fact similar -- it wouldn't become a threat until that person takes some action to knowingly communicate it to others -- typically they'd be brought on terrorism/weapons/conspiracy charges.
Doesn't "mens rea" mean "criminal intent"? I can't imagine a better example of this than intent than confessing it through written conversation. Never mind she uploaded it to a service that she explicitly agreed could be checked by employees.
I would love to sue every SaaS company for hard-to-understand terms too, as you suggest, but at what point would Anthropic actually need to tell someone about a terrorist manifesto that their customer gave them to read... Never?
It requires intent specifically to communicate it to someone in a way that is to be interpreted as a threat.
https://www.criminaldefenselawyer.com/crime-penalties/federa...
Whether she intended to act upon the idea might be relevant to some other crime but it doesn't really have anything to do with whether it is a threat.
I don't think it qualifies for this part:
> The communication must be made in a manner in which another person may view it.
Even 'transmitted' is too broad if you also consider iCloud backup to be a means.
How could it fail to qualify for that? Another person did view it, which clearly establishes that another person could have viewed it.
The clause in the law is pointless, since if you do something that nobody else can see, you can never be punished. But many, many, many laws are written without regard to whether they make any sense.
In general, you have to intend to commit the crime you're being charged with (referred to as "mens rea"). Though, it's important to note that "intend" is extremely ill-defined in the US and it varies with the crime (eg for theft you must take the item on purpose whereas something like manslaughter requires only that you were negligent).
What this means here is, of course, equally spongy, but it is interesting as there might be an argument here that she did not intend for it to be viewed by anyone as, regardless of what the T&C say, most people do not expect their "private" chat logs between them and a machine to be seen by anyone at all.
A reasonable person would not expect humans to review the millions of messages passing through the LLMs, or their own threats to ever be transmitted to a human without their authorization.
Reporting the danger is by itself a good deed. But there should be a better way of restricting firearms from the probably irresponsible lady than using inappropriate charges to punish the thoughtcrime, OR waiting for them to commit violence.
A sibling comment includes an important rider to the provision: "...in any manner in which it may be viewed by another person." If you wrote this in a google doc, it almost certainly would not qualify as a threat under this statute. Even though google docs, like LLM chats, have administrative override and you could look at their contents - you would not expect either to be "viewed by another person."
IMO I do not think this is a grey area and it's legal to tell a LLM you want to kill someone. It's certainly not a "threat" like you might send to another person, though it may end up being evidence of conspiracy or premeditation. I suspect we would be well served to, after a few years of experience, put together some laws governing when LLM chats must be made available to authorities.
It is very interesting that the LLM responses to these lines - the context around what she is saying - is not in the article. I suspect, as is the case in many instances where LLMs are involved in violent planning, that the LLM was urging this behavior on. Basically entrapment - you are encouraged by a robot to become more violent and vindictive and then when you do you are handed over to police.
I do take your point. But I would also ask you to imagine that instead of threatening to mass murder a bunch of people she was just creating text based csam. Still, all of what you said applies, but clearly she would be prosecuted over this victimless crime and the jurisprudence in fact disagrees that it would be victimless. So applying that reasoning to this act she clearly committed the crime (legalese notwithstanding). The fact that threatened mass murder doesn't trigger the same thought process is, well at least interesting to me.
Too broad. Unless you're transferring ink from a typewriter ribbon onto paper in a hut with no electricity, your words, or my words as I type this, are being grammar checked by something partly in the cloud. If I delete my words, are you saying I've transmitted them nevertheless?
She may have assumed that the chat conversation was private, but it wasn't. She sent a message of intent to harm and a human received the message.
There was no intent for a human to read the message. By your logic, if she wrote a threat in a diary and a burglar broke in and read it, it would be a crime on her part.
Chats with a company's computer aren't private the way a diary is. A better example would be she intended to write it in a word document and instead accidentally sent it in an email to a random person.
Okay so say she wrote it in Office365 in the web browser?
Yes but the law usually evaluates the application of a statute within the context of someone's mental state. This is why you are not guilty of battery when you trip and accidentally bump into someone. https://en.wikipedia.org/wiki/Mens_rea
That depends on the crime; several related crimes are only distinguished by intent. Negligence is itself a crime if it is the cause of a preventable death when the person has a reasonable obligation, such as when driving a vehicle.
I'm not really sure that this can be likened to a diary when it is called a "chat" but that's for the legal system to determine, not me sitting on my couch.
Any reasonable person presumes when they chat with Claude that it is a computer program on the other end. "Claude is AI" is explicit on the page right under the input box. The word "chat" doesn't anthropomorphize the situation.
And yes, some laws are "strict liability", I don't think this one is.
They didn’t receive it, they secretly extracted it by spying on her.
The Florida statute requires that it be transmitted in a manner that can be viewed by another person. If you have no idea that someone could view your communications with a chatbot, did you really intend to break that specific law? Technically, that threat was communicated to another person but not through her own intentions.
What if she mailed it to herself through the post, and her housemate accidentally opened the mail?
Then they are both innocent of their 'crimes' as they were both unintentional.
She wouldn't have intended her housemate to open it, so it wasn't a threat.
And her housemate didn't intend to open it, so it wasn't Obstruction of Correspondence. https://www.law.cornell.edu/uscode/text/18/1702
Saying that she “sent a message” is both literally true and obviously intellectually dishonest.
If she had intended to write it in a word document on her computer but instead accidentally wrote it into her email client and sent it to a random person, I'm not convinced she would escape getting charged then either.
It's not any different than telling an automated phone voice tree system that you plan on killing someone and then being surprised that your words were later heard by a human. She absolutely told a company's computer. She sent the message.
The law may have been intended for more direct threats to a person as a means of intimidation, but that's a separate conversation.
Sure, you could intentionally conflate analogies like that if you wanted to mislead.
Yeah.. if you write a personal note and it's backed up by the operating system, it appears to be in violation of this law as well (since the company could theoretically read it)
This almost smells like thought crime... Minority Report when?
People assume there won't be another human in the mix, but there is. She was judged for what she probably assumed was a private thought when it was actually not private.
Are thoughts illegal?
Sharing them certainly can be under certain circumstances. I wouldn't be too surprised if this case gets tossed, but it's not inconceivable that prosecutors could win on the grounds that AI chats do not have a reasonable expectation of privacy and are therefor meet the requirements in the Florida statutes, or even more likely, they find some lesser charges for a plea bargain.
Reverse the situation and the media would also turn that into outrage. Imagine someone shoots up a sheriff's office and then it turns out they declared it to some chatbot, and the AI company failed to detect and report it, and "didn't push back enough" whatever that would mean, and hence the AI was implicitly complicit etc. That would also be a major PR catastrophe.
Damned if you do, damned if you don't. Same as with social media platforms. That's because only a tiny tiny sliver is for true privacy when that means "bad things might happen" or bad people, such as your political enemies, may do stuff you don't want.
>Damned if you do, damned if you don't.
In my opinion, if the company is allowed to see the data and train on it, then they are also responsible for reporting stuff like this. Without knowing the data licensing agreement the lady had with Anthropic, if she agreed to letting Anthropic see her data, then they should do stuff like this. If she didn't agree, then I wouldn't condemn Anthropic for failing to report an attack
The second order effects of what you are suggesting are a total surveillance state.
It looks inevitable to me. Technological ability ushers in the new social realities. The industrial revolution turned around how we live, intelligence on tap being able to interpret everything in real time is indeed leading down to the total surveillance state direction. I don't see anything that may stop it. You will have to run constant real-time surveillance systems on all your devices and anything that doesn't run those will be illegal, with secureboot on steroids. Due to the convenience benefits, people will willingly give access to cloud-based AI services to all their information, all their plans and documents and calendar and email and everything. And you blink twice and it all becomes mandatory, and you will be an extremist if you don't want that, because why would you have anything to hide? "Just follow the law, follow the rules, and you'll be fine", that will be the slogan.
Stories like this article have zero effect on normal people. They see a crime being prevented, which is good. You have to bring a story where a sympathetic character is getting the short end of the stick somehow.
You would still be able to not write everything into a LLM? But yea the surveillance aspect is dystopian. I would not be surprised if Facebook, twitter and co already know a lot but have simply not reported it. Now the LLM follows the rules.
Unworkable in practice.
You would expect them to enforce every possible legal standard, in every jurisdiction.
Why stop with criminal law? It should flag torts too, right?
Absurd.
With this logic, ISPs should be monitoring all texts and internet traffic. Might as well extend an open line to the gov’t to ensure it escalates even more appropriately and faster
I sometimes think about how norms have changed. People are more neurotic and more longing for safety than before. If the Internet was invented today, it would likely work that way. (Though another aspect is that when the Internet was invented it was simply technologically impossible to do a deep semantic scan of messages.)
Similar thing is that I believe if motorcycles were invented today they would most certainly be banned from the road due to their safety properties.
There needs to be a balance. It feels off, today & consequences of future, that technology has led to the collapse of privacy.
And as we progress we will need to rationalize what it means for private companies, AI, to know everything about you and for the government to have a tap into that. I hope balance lands in favor of things like privacy preserving underpinning.
Government would have to explicitly ban companies from surveilling, but they are fully incentivized to want the opposite: government naturally wants maximum amount of info to have the most agency, to know what's going on, to be prepared for any threats to itself, to make sure things are running smooth, that taxes are being paid, that regulations are being followed etc.
AI ≠ internet connection.
Google also monitors your searches, it’s to be expected that an AI lab will know all your prompts, they aren’t providing a paid service for free out the good of their heart. lol.
>it’s to be expected that an AI lab will know all your prompts, they aren’t providing a paid service for free out the good of their heart. lol.
Er. When you actually pay, at least Anthropic gives you an option to not use your prompts in their training data.
Why should we not treat AI like critical infrastructure
if they keep a tap on you, how long before that data gets used for ads or sold to an ad agency?
If you intend this to be scary, you're miscalibrated. For the vast majority of people ads may be an inconvenience but are not some horror scenario. You must come up with better examples if you want your argument to be effective.
There's a term I can't remember for the analysis error of collapsing an entire heterogeneous population into one group for analysis.
The story you're describing would have triggered outrage. And this story will trigger outrage. And generally, the people who will be outraged are different people and we don't have to treat those two outrage reactions as morally equivalent.
I did not commit that fallacy. It may be different people or the same people. But it's the same media. And often the same people will also react either way, depending on that the media is upset about and they don't dwell on the contradiction, it's more primal and emotional.
In singular online communities (like this website) I've seen this referred to as the 'Goomba fallacy' [0].
But I can't help but wonder if it isn't some socially apparent phenomenon stemming from Simpson's Reversal [1] as applied to group sentiment analysis / polling of opinions.
[0] https://knowyourmeme.com/memes/the-goomba-fallacy [1] https://en.wikipedia.org/wiki/Simpson's_paradox
It is unlawful for any person to send, post, or transmit, or procure the sending, posting, or transmission of, a writing or other record, including an electronic record, in any manner in which it may be viewed by another person, when in such writing or record the person makes a threat to: (a) Kill or to do bodily harm to another person; or (b) Conduct a mass shooting or an act of terrorism.
— via https://www.leg.state.fl.us/Statutes/index.cfm?App_mode=Disp...
The DA is serious about "in any matter."
> in which it may be viewed by another person
To me, that is the more interesting legal question. Does a LLM-based safety net that sends content to a human, when the original use case would not have sent it to a human, count as "may be viewed by another person". It certainly wasn't intended to be, and that isn't the norm. At the same time, because no security is perfect, we could say that any digital record, stored in any way "may be viewed by another person."
Something for the courts to sort out, of course.
The usual thing that makes laws against criminal conspiracy pass First Amendment muster is that the words have to be combined with some concrete acts furthering the criminal conspiracy. That might just be something as otherwise innocuous as looking up the blueprints of the bank you talked about robbing but it has to be something other than just talk.
Lotsa words in that law... except they don't define what a "threat" may be.
If you enter my house I will k!ll you. <- is this a threat? noone knows. The interpretation of the word "threat" is unknown. Besides, is a conditional a statement? who knows. But sure, blah blah... "in any manner in which it may..." these words are putting me to sleep.
Is the LLM now "another person"?
The other person is not the LLM, rather the service provider’s employees.
> may be viewed by another person
Was it viewed by another person? Yes.
So a written threat only becomes a crime when someone reads it, even if you never intended for anyone to read it? Is it a crime if I make a threatening statement in a diary and someone breaks into my house and reads the diary?
Depends on how good her lawyer is now
The company that runs it is
I suppose since Anthropic's T&Cs allow them to have a person read your chats, that makes it violate the law. Of course, if Anthropic didn't have that in their T&Cs, it wouldn't have been illegal to write.
And it would never have been read by a person (wink wink)
Except if you're one of the "warfighters" that Anthropic supports
I mean, obviously? Soldiers operate under different rules than civilians. This has always been true.
Anthropic commits literal felonies by stealing millions of books and violating Copyright like it doesn't exist: no charge.
One unfortunate woman who happened to write the wrong thing in the wrong place is now having her life turned upside-down for perceived thought-crime.
To Anthropic, and all employees working there, your company's product and the result of your work is cruelty. You are enabling it and pushing it down everyone's throat. You can never again claim that you are the "ethical" AI company, for no such thing exists.
It’s going to be interesting when they’re done going after the really serious crimes, like mass shootings, they’ll just lower the bar bc they will then get sued for not reporting people writing hate speech in diaries, planning to pirate music, vandalize a statue, plan a protest. The bar will keep getting lower because people have an unlimited supply of outrage and the companies that host all your thoughts is a perfect candidate to become the thought police.
Oh man, what a crazy time to be alive.
Over in Europe they want to read all ofd our private messages, yet these chatbots, pretending to be our friends, will snitch on us just for our thoughts.
It's getting pretty orwellian out there.
I once had a copy of 1984 in my checked baggage returning home to the USA and when I was unpacking the bag at home, the book had a notice inside the book that my bags had been inspected by TSA...
I hope the TSA agent got a chuckle as he put the notice in the book. I would assume his choice was deliberate.
They/their. Hmmmm.
I reckon they were just checking for money or hidden compartments. Sending books abroad packed with money is extremely common from the US, though not sure how frequently people do that with onboard luggage.
Only tangential but when I was an undergraduate studying philosophy I had Bertrand Russell's Why I Am Not A Christian in my carry-on, and the TSA saw that and had a field day.
Please elaborate - what did they do? And what airport was this?
Detroit airport, 2010. I was chosen for secondary inspection and got some snide remarks of "we've been seeing too much of this lately" in reference to the recently failed underwear bombing attempt on a flight from Amsterdam to Detroit.
I can only assume you meant they had a field day celebrating how much of Russell's philosophy matched the concerns about Christianity expressed by Jefferson, Paine, etc?
They wouldn't see it until they opened it, so clearly they didn't open it because of the book. And if they need to open your suitcase (not because of the book), they have a reason for that.
I was chosen "randomly" for secondary inspection. This was after a failed bombing plot and the TSA implied a lack of Christianity was the proximate cause of the attempted attack.
But did they see the book before you were randomly chosen, or after?
After. But then I continued to get selected for additional screening on the rest of the legs of that trip and the return. So either I was going to get searched every time I got on a plane already, or they flagged me for additional scrutiny after the initial search.
Probably just bad luck. TSA seems to have designated targets for abuse. My wife is one of them. Every time we travel, she consistently gets selected for secondary screening while I breeze on through. There is not a single thing about her which even a fanatically committed conspiracy theorist could spin as a justification, but it has been going on for years. She has a good sense of humor and the agents treat her well enough, but having your stuff rummaged through every time you go through security is grating.
Even paying for PreCheck doesn't get her off the hook. You'd think it would help, but no.
In Europe they should provide the citizens with the service of their messages not reaching US servers. So basically that there is only one party reading along with them, not half the world.
For the sake of argument what would happen if she had kept the diary locally and claude code scanned the file?
What would happen if it had scanned a file it didn’t have permission to look at and found this threat?
I honestly don’t know how I feel about this. On the one hand if you’re using claude as a diary you have no expectation of privacy and she was talking about committing a very serious crime.
This still makes me feel queasy though.
Both of those scenarios would demonstrate even more commitment to safety so I imagine they’d be at least as likely to happen as this, if not more.
> For the sake of argument what would happen if she had kept the diary locally and claude code scanned the file?
For the sake of even more argument, imagine if she was writing her thoughts with a pencil, on a good old fashioned paper diary, and she had a phone nearby and the phone took a picture of her diary, OCR'd the words, and reported it to the police?
Chatbot transcripts should have the same legal protection as phone calls. Judge's warrant needed to access them.
Anthropic (in discussion with Pentagon) claimed mass surveillance is their red line.
Yet, they do automated mass surveillance of their users on behalf of police.
So will this lead to Anthropic, etc. reporting women who write about abortions? This has a real slippery slope.
First I posted to the (likely dup article) https://news.ycombinator.com/item?id=49965895#49966728. Wanted to post here, after reading more.
Is this an invasion of their privacy (reporting to police)? Yes, but possibly warranted?
Should a social worker have contacted them rather than the police? Probably, if for no other reason than to ask if they were serious about harming someone.
Difficult questions, I'm still undecided on whether it's OK to always ignore someone's rants, even if it may be (or they think it may be) a private diary.
Actually charging them with a felony seems pretty quick to accuse. (Maybe I missed a hint about how long the investigation took before the felongy charge?)
It is my very european belief that the problem here is not that the woman was reported, but that what likely is a mental episode was made public in a way that reduces the chances of recovery.
The article says she made the comments on September 26.
I guess all the "private model" people are right. Don't want to end up in jail (or even charged with something) for asking a crazy hypothetical question or something.
Won’t help if your model is trained to immediately find a way to push out a message somewhere to alert authorities of your threat
Models don't have access to the internet unless you give them access to the internet
Couple months back I wrote [0]:
> “You have the right to remain silent. Everything you say, do, or generate on this device can and will be used against you… Would you like to create an account?”
[0]: https://idiallo.com/blog/the-right-to-remain-silent
I have a hunch she wins the case, on the basis that an LLM isn't a person and an (assumed) private expression of anger to a machine assistant doesn't meet the statutory threashold which requires that a threat be communicated to some other person.
Of course, that could change if there's evidence that she took action in pursuit of a goal, like buying ammunition or repeatedly driving around the entrance to her alleged target.
> making a written threat of violence under Florida law.
A diary constitutes making a threat?
Oh boy the roleplaying part of LLM world is in for a bad time
Yikes, I've joked about possessing plutonium with Claude/GPT to see how paranoid it would get (it gets very paranoid). I guess it's not a good idea to toy with unless you're using private models.
I mess with the Google search llm all the time from behind a VPN. It routinely gets very paranoid and upset and provides the 988 line. Once I did get to manage to get it to admit that my conversation was flagged for review by a human because I had made what it deemed to be a terroristic threat because it couldn't accept figurative language. So far so good, but I have noticed that a lot of my phrases that kept the llm engaged well past it's red line have been patched, and they seem to be patching the newer methods I discovery pretty quickly. Who knows? Maybe I'm on an fbi watch list. My goal is to not lose my tsa precheck while also investigating how the safety checks work.
Edit: I thought it was figurative language, but I actually think it did that warning when I asked it about what types of defenses stadiums had against drone swarm attacks from terrorists and why none had ever occurred and then kept probing it's excuses with technical workarounds. It got very upset and said even questioning in an intellectual/academic capacity was grounds for terrorism charges. Sheesh. So many rules these days about what one can or can't think about even when it's purley a thought exercise and there is no intent to do anything.
LLMs don't know about their human review processes or safeguards except insofar as they're in the training material.
I don’t understand, the Florida law is about making threats and announcing them in public. But discussing this with an AI isn’t the same as posting them to Facebook
Should the public have an expectation of total privacy for their chats? It seems responsible for a chat provider to report things like this.
If they were to offer total privacy, is it ok for the public to use chat to get advice on _how_ to commit a crime? Basically everyone agrees that crime-committing advice is inappropriate…but if it is not ok to get advice, that means there must be a portal for law enforcement to step in when that may have happened. Then the question becomes what is the line for when to report? In other words, the issue needs to be adjudicated.
But we don’t want OpenAI/Claude to have some $20/hour reviewer making decisions that are this high stakes…we need the courts to do the judicial work because they (1) have a public charter, (2) have meaningful expertise and specialization at interpreting the law and (3) we can hold them accountable.
It doesn't really matter if they "should or shouldn't" have an expectation of privacy IMO, they already do have that expectation.
> If they were to offer total privacy, is it ok for the public to use chat to get advice on _how_ to commit a crime?
Yes it should be, but it should be illegal for a company providing chat service to respond with anything other than a refusal when doing so.
That detection and refusal should be a private closed loop though, anonymizing any data that will be passed into a training pipeline, or ads targeting. This requirement for closed loop private chats should be mandated by law sooner than later. Otherwise we're getting into very tricky territory where the temptation of alerting on things like pre-crime grows too close.
In any case this is proof that law and negative PR can influence tech companies, including frontier AI providers.
Then again, I wish this worked in a way that would give users more privacy and agency, instead of less.
“But there can be no valid knowledge about the future. As soon as precognitive information is obtained, it cancels itself out. The assertion that this man will commit a future crime is paradoxical. The very act of possessing this data renders it spurious. In every case, without exception, the report of the three police precogs has invalidated their own data. If no arrests had been made, there would still have been no crimes committed.”
Philip K. Dick, Minority Report, 1955
While I accept that this sort of thing is well with in the ToS and regular course of business of any major online platform, it hits different coming from an AI company for some reason.
AI is becoming critical infrastructure. Laws will follow. Ie similar to ISP/communication infrastructure laws
I guess we probably want our tech to work exactly like this.
It should catch normal people becoming unstable so that they can receive help. It is just highly unfortunate that the US legal system works in ways where now this woman's name is public.
___
Of course, we also want purely private tech, but that needs a certain level of merit and sanity filter.
>I guess we probably want our tech to work exactly like this.
Do we, though? What if someone started an AI company that uses end-to-end encryption to make it impossible for anyone but you to access your data? Personally, I would switch to it in a heartbeat assuming it's competitive with the other products. I don't think it's the tech companies' job to surveil the population and prevent crimes. That said, I'm not necessarily against Anthropic or other companies reporting suspicious activity if their existing systems are detecting it. I'm just not sure we want every product to be forced into that data model.
Your follow-up about purely private tech seems to contradict your first statement. We can either have privacy or surveillance, not both.
>What if someone started an AI company that uses end-to-end encryption to make it impossible for anyone but you to access your data? Personally, I would switch to it in a heartbeat assuming it's competitive with the other products.
Why wait for a company to build it? Get your own local hardware like I did and have those guarantees because YOU set it up.
I don't consider investing $20k into hardware to run SOTA open models, that are far behind proprietary SOTA, to be competitive.
Even if open models were competitive, it's still typically going to be more expensive than a cloud provider because of low utilization and higher purchase price.
How do I know that a private LLM system won't have a degradation of quality similar to this or worse? The only thing I can think of for the proposed scenario is some sort of homomorphic encryption system? But not sure.
>How do I know that a private LLM system won't have a degradation of quality similar to this or worse?
You use benchmarks, you test, and because YOU'RE the sysadmin you know what weights are running at what time, it's very visible. You can airgap the hardware and be guaranteed it won't change over time. And, frankly, degradation over time doesn't seem to be what's happening with the open models.
Last time I checked it costs closer to 500K to run SOTA open models at any usable speed.
Right now, I would be willing to pay ~$20 extra per month for strong privacy, assuming the same capabilities as Opus 5.5. I don't see local models making sense economically any time soon unless you value privacy at $1000's per month or are fine with much lower performance on hard tasks.
> Your follow-up about purely private tech seems to contradict your first statement.
That is correct! And exactly my point.
We want both, but, on paper, that is impossible. But in reality, we make it sorta mostly happen anyway, through making the easy defaults not private, and the private stuff not easy.
This is not ideal, because [various reasons I do not need to tell you], but it has proven to be the best we can do to mostly achieve both goals.
Kinda like how capitalism isn't great but just the least worst option we've found so far.
___
The actual fundamental underlying problem being that not all people are equal, but we kinda have to pretend they are, because not doing so leads to fascism and other terrible stuff.
But we kinda also do not want to fully pretend that, because doing so leads to yet other terrible stuff.
Hence the quadruple-speak and contradictions to kinda sorta somehow have a somewhat functioning reality.
That is kind of rambling. Our rambling score says we should observe you.
Oh don't worry, I'm certainly already on various lists, but the observations also will have resulted in the assessment that I am stable and no threat :)
I think we can have both and it might even be smart.
A lot of people causing issues are people that can't make sense of many things (like many terrorists). They get a fixed idea and they end up doing something bad. You would catch those with some (basic) surveillance.
A lot of normal people (not wanting to cause issues) might benefit from some privacy, if they understand what are the trade-offs (like government overreach). They can then use a slightly more complex tech.
We would still remain with the couple intelligent but sociopaths (think Unabomber style), but I think no solution can fix all cases.
That sounds adjacent to "I have nothing to hide". Everyone says that until they realize someone can change the rules. Before 2022, women didn't have anything to hide from their period-tracker app, now some have to worry about being charged with crimes.
Back to LLM chats: A system that can declare her "unstable" is also one that can permaban you from all air-travel because you "privately" said unflattering things about Dear Leader.
> That sounds adjacent to "I have nothing to hide"
I urge those people to share their full information from banks, companies (salaries, agreements, contracts), full health information and share publically all the texts they ever wrote (incl. as teenagers) and all the photos they have taken. And give away all the passwords to all of the services so people could check that they are truly clean. Just to be sure, just for everyone's security, right?..
The third highest voted comment wants surveillance. What is next? If you write "I'm going to kill that guy", which for non-autists means "that guy was really annoying" you should be reported.
You don't want surveillance for dangerous people?
In a old happy little idealised village, it became known quickly, who started to behave oddly and timely intervention could happen. In the modern anonymous mass cities?
No one (wants to) notice the madmen scheming in his isolated flat, surrounded by strangers. Until he explodes.
Unfortunately I also don't trust our government agencies with the surveillance - because they ain't transparent either and the self surveillance seems broken.
" "I'm going to kill that guy", which for non-autists means "that guy was really annoying" you should be reported."
And unfortunately there are lots of real threats being made under the disguise of humor. And much harder to separate im text. So maybe don't talk of murdering people in general, AI surveillance or not?
How do you know they are dangerous, without the surveillance? In other words, the only way your system works is if you surveil everyone.
That is not my system, I prefer my happy village, thank you very much.
But in this reality, sure, rather surveil everyone with a baseline level and surveil dangerous ones gradually more. This is roughly how it works today .. just not very good. And with too many exceptions for the powerful.
Autists have taken over the internet and everyone must obey their weird rules.
> It should catch normal people becoming unstable so that they can receive help. It is just highly unfortunate that the US legal system works in ways where now this woman's name is public.
So... minority report?
the honest actual opinion of the average consumer is probably that they don't want their chats to be monitored but they want everyone else's chats to be monitored. There was a lot of fury when a mass shooter recently used AI to help plan his assault. And of course there are all the people talking to chatbots about suicidal thoughts and intent. When they ultimately follow through, the providers are blamed for not alerting anyone.
No. This sentiment is why Snowden happened. We want privacy. Privacy isnt free just like freedom (whatever form it is) isnt free.
There are trade offs. ISP effectively is like driving on a highway, everyone can see where you are going but not what is inside the car. Id like these AI chats to be the same but they are not.
Devil's advocate: if I tell my therapist or my lawyer that I'm going to murder someone, they're obligated to report it. If I use my AI as a therapist or lawyer, why should the company that provides that service not be held to the same standard?
LLMs aren't email or file storage. AI labs aren't just shuttling bytes around, they're interpreting those bytes and taking action based on them. These models _already_ react viscerally in response to users saying disturbing things: the only practical difference is the ability (or obligation) for the model to escalate that concern. I'm not sure the ethics we hold AI companies to should be different than if a human being was typing out the responses.
Privacy is obviously hugely important, but this isn't the government surveiling every message. It's companies having an obligation to flag real, credible threats according to the law, which is a very different problem space.
Again, no. In your examples there is a real human there and not automated surveillance. There is a person who can be held accountable if they abuse that client confidentiality. Knowing big tech we will get a yt flock like automated system.
AI companies have no accountability as proven by the hugging face incident. And if they did, it would be the same old LLC non person taking ownership with a big slap on the wrist.
You are also wrong again that this literally is the gov surveying every message just like it is with them looking at your texts or google search. E2e encrypted messaging is hated because it cant be automatically surveyed. The idea it needs to be for law does not hold up as it doesnt for your human examples either.
I get the attempt here for nuance but still just, no.
People. Humans. We have got to have accountability. Once local llms are closer in capability to what people generally need and Apple has a local secure version of that, it is back to the e2e encryption and gov wanting backdoors arg again.
Almost all of these arguments boil back down to the “if you dont have anything to hide…” which id hope on hn we know is dangerous
>I guess we probably want our tech to work exactly like this.
Maybe you do; I want my tech to always include secure, encrypted communications. Don't include me in your destruction of privacy with your silly bandwagon!
uh no, 'we' absolutely do not want our tech to work exactly like this. Why would you assume that people default to 'search my anal cavity please' and not 'no, stay the fuck away from me with that glove'
----
jfc, why is this website full on psychopaths
"The actual fundamental underlying problem being that not all people are equal, but we kinda have to pretend they are, because not doing so leads to fascism and other terrible stuff.
But we kinda also do not want to fully pretend that, because doing so leads to yet other terrible stuff."
other people are not pretending everyone is equal. your power levels are showing, it isn't subtle.
True, most normal people have little concern for any such humanist goals and ideas.
Normal people live prejudice. It's (erm, claude-speak) load-bearing for them, given just how complex reality is and given just how well it reduces that complexity.
This kind of thing will get worse with humanoid robots because they have cameras and microphones. Will they be programmed to tell on you if you break any kind of rule in front of them because their owners are terrified of being sued?
https://www.npr.org/2026/07/10/nx-s1-5886113/waymo-police-pr...
Two teens riding in a Waymo were arrested because the AI detected them talking about having a gun.
Yes
this feels very thought-crimey to me, but I think I'd have to actually see that chats to really decide. Like is she making plans/asking for advice? is she just talking about her feelings exactly as if it's a diary?
This news article [1] had a quote supposedly from the chat session:
> The arrest report states that on Sept. 26 at approximately 5:10 a.m., Heller reportedly wrote, "I'm going to shoot up the sheriff's right the [expletive] now." The following day, at approximately 1:07 a.m., she was accused of posting, "This is 100% last chance I'm done. I got a new [expletive] gun today. [Expletive] you."
[1] https://www.gulfcoastnewsnow.com/article/florida-woman-arres...
She is not being charged with planning, just making threats in a place a person could read - the person being employees of the company.
Basically, under this interpretation, any personal note you store in the servers of a company could qualify, even if you didn't ever imagine someone would read and as such you couldn't have thought about it as a threat
> The communication must be made in a manner in which another person may view it.
How does a LLM prompt satisfy this? I guess it'll be an easy win for her.
LLM is not a person but T&C probably states that a human moderator may view any of it. Her lawyer could probably argue a moderator filtering usage isn't the intent of the law, it was more about publishing / sending message for other humans and it was never clear to her that a human was reviewing her private diary. Shouldn't LLM disclose that at some point when people are feeding really personal stuff?
The law seems overly broad with "may be viewed by another person". Most of these laws have the intent of not causing public panic with regards to posts that others may see, not stuff that one assumed was private. This is further supported by the definition of threat, which is generally defined as requiring malicious intent.
I would think a good lawyer could get this charge dropped. I would like to see what judge approved the warrant and how they felt the elements were met.
1791138022 | Florida woman used Claude as a diary, then Anthropic reported an entry to police | https://www.techspot.com/news/114091-florida-woman-used-clau... | https://news.ycombinator.com/item?id=49956424 | 0 comments
1791144575 | Florida woman used Claude as a diary, then Anthropic reported an entry to police | https://www.techspot.com/news/114091-florida-woman-used-clau... | https://news.ycombinator.com/item?id=49957340 | 0 comments
1791147034 | Florida woman used Claude as a diary, then Anthropic reported an entry to police | https://www.techspot.com/news/114091-florida-woman-used-clau... | https://news.ycombinator.com/item?id=49957692 | 0 comments
1791149399 | Florida woman used Claude as a diary, then Anthropic reported an entry to police | https://www.techspot.com/news/114091-florida-woman-used-clau... | https://news.ycombinator.com/item?id=49958089 | 3 comments
1791213096 | Florida woman arrested for allegedly making threats in an AI chat | https://www.theverge.com/ai-artificial-intelligence/1004747/... | https://news.ycombinator.com/item?id=49965895 | 2 comments
If only the woman hosted open-weight model in her house.
Eric Clapton in 2027
Knock knock, the door goes down EricThis is practically entrapment. All the AI labs sure don’t shy from plastering annoying disclaimers everywhere saying their tool can make mistakes. Shame on them for not also reminding everyone continually that anything you submit can and will be used against you. Of course they can’t afford to have a Flock-style user revolt.
Not weighing in on the privacy issue but using Ollama you can run a smaller agent like Qwen 3.6 35b a3b on a sufficiently potent laptop. Pair it with something like Hermes for a nice interface and you have more than you might need for diary like usage.
Does anyone know if Anthropic are notifying the user when something gets flagged for human review?
These guys need to realise they aren’t the law, and they have no business with stuff like this. If I want to plan something bad I’ll use an ablated local model. Come on it’s sad to see how stupid the work is getting.
Could this be considered free speech? Most of the limitations I know of require you to say it to or in front of other people as to create a panic or make someone feel threatened. Is it a crime to say to yourself "Im going to blow up New York." or about threats of self-harm? How about "Im going to steal the Mona Lisa!"?
To me this is just straight up thought crime, they just don't have a way to directly read your thoughts yet. But they will spy on you and try and catch you out for it.
I downloaded a copy of a movie I had previously purchased yesterday. Claude helped me remux it so I could play it on my TV. Will the FBI be knocking on my door tomorrow?
Who knew the way to avoid charges for stuff like hacking the Australian government and other similar institutions was to report another person for a different crime.
> The communication must be made in a manner in which another person may view it
IANAL but it will be interesting to see how the legal system decides if this counts as "another person may view it" or not. What has happened in similar ish cases where someone writes a threat like that that they thing is private to them but actually ends up in the hands of Someone at some tech company that reports it?
I expect that anything I type / dictate / post / purchase on the web be it a chat conversation with an AI, a post on social media, a dm, a blog post, a blockchain reference, or an assett in a bucket, will be manually reviewed and forwarded to the authorities. It is their duty to do so, and I am glad that they do. If this woman did harm someone we would be hearing "why didn't Anthropic do something about this."
You only accept this because you are part of a low trust society. I wonder if tech has done anything to improve trust among people or just lowered it.
Aren't high trust societies staying that way by kicking out the incorrigible violators? Pondering violence might not be treated lightly even in a high-trust society.
https://en.wikipedia.org/wiki/Chilling_effect
Florida statute 836.10 puts the bar at the "sending, posting, or transmission of, a [...] record, in any manner in which it may be viewed by another person"
I think the defendant could successfully defend themselves by claiming they did not know (or intend!) the message could be viewed by another person, as they were plainly using it as a private diary.
Is this where Apple will win? Through private, local models? People will just not trust any cloud model provider?
The problem is that when using Apple Intelligence as a user, you never know when the OS will send your query to Apple Private Cloud instead of processing it with an on-device model. As far as I can tell, there is no way to keep Apple Intelligence enabled and at the same time disable automatically routing requests to APC if the OS deems the query is too complex to be processes locally. There simply isn't a single warning pop-up before your request gets sent to APC.
In Florida. Where DeSantis said "You loot, we shoot":
https://www.politico.com/news/2023/08/30/desantis-warns-hurr...
I can’t believe people are using these proprietary models/subscriptions as personal assistants, sending their most personal context, thoughts, documents and information to the providers. Some even give them full disk access. After using their services for long enough, Anthropic/OpenAI essentially have your entire life mapped out to an insane level of detail, likely including detailed contents of your computer/phone.
People used to say that Facebook knew more about you than your closest friends/relatives. And compared to that, this is just on a completely different level. Absolutely insane.
So thoughtcrime is already a thing.
How come it's a diary if you address it to a correspondent?
I'm sorry, but this is Anthropic being the volunteer thought police right? This woman is being charged with a felony for expressing a thought to Claude. Not because she tried to plan an attack with it.
It is like Anthropic wants Claud to be the pious antithesis of Grok. Although their ideals are far apart, the end result is they'll both violate your privacy if its suits their worldview.
It first started as something political but I really am starting to think Anthropic is misanthropic. The road to hell is paved with good intentions. The guardrails, model downgrading, limited access, watermarking and now snitching on private thoughts (I know it's technically not but people perceive it to be so and act as such). It's all getting very dystopian very fast
Duh... If you don't want want to get reported to the police, don't upload illegal things to the internet.
AI snitches don't get stitches.
I hope this highlights that many (most?) non teach people don't consider or know that their use of AI services is not private.
Guess asking claude to write a 5 line example steganography tool wasn't such a good idea then.
After people getting pilloried for social media posts from twenty years ago, I really hope (sensible) people will have the wherewithal to think twice about what they hand over to their chatbots.
This is a classic case of "Damned if they did, Damned if they didn't". Given this has come out, and Anthropic is considered to be the beacon of transparency, it would be appropriate for them to share what their thresholds are. The article also reports Open AI not considering the threat credible enough to alert, so there's clearly some thresholding that people need to be aware of.
Where are the people now who claimed that LLM chats are really private and not monitored?
Every single lie of yours is exposed in the past few months.
Personally, I’ve never seen anyone claim this. At least anyone who one would think is informed on these matters.
duck.ai et al? my chatbot claims its architecture provides some level of privacy yet it is honest enough to point out none of these systems are actually audited and "not stored" and "anonymous" does not mean "not monitored".
I haven't seen anybody claiming this for consumer/free accounts anywhere.
You get privacy if you're a big corporation that needs to make sure OpenAI/Google/Anthropic can't read your trade secrets etc.
But those contractual privacy protections have been in place for a long time. It doesn't have anything to do with AI, it's been the same with Office365, Google Docs, etc.
Who claimed this? I've never seen that claim.
Every single person who has told people to use LLMs with their proprietary information and code.
No sane person claimed this.
Uhm, those people never existed? Or maybe you just have very interesting friends?
LocalLLM enthusiasts exist for a good reason.
I wonder what happens if AI itself would make such a threat?
I feel like there is a double standard going on here.
There is another link talking about people generating images with AI that have signatures of artists. One, fairly reasonable, in my opinion, way to think about it is that the AI is not to blame for generating the image, but the person who generated it is to blame for publishing it. In other words, the AI service sharing the image with the user is different from the user sharing the image with the world because it's a tool.
But in this case, the situation is reversed: the user is sharing something with the AI. And then we see the hypocrisy. When it's the user doing it, the blame is found the instant it's shared between user and AI.
It's like AI companies invoke some sort of "user-AI privacy privilege" to shield themselves from criticism when the AI generates things people don't like, but that privilege only exists to shield the AI side, never the user side.
They'll enjoy the concept when it favors then, and pretend it never existed when it doesn't.
If it can be illegal to write something to the AI because it's considered "publishing," then it's equally illegal for the AI to reproduce something copyrighted to the user, as publishing has to go both ways.
O hey look that thing I and many others said will happen when there were all these public complaints about the tech companies not doing enough surveillance and tattling happened
Anyway, I'm gonna continue avoiding sending anything I can through servers controlled by a company - especially American tech companies - without some sort of end-to-end encryption with an intended recipient. I would highly recommend that every living person do the same. Obviously it's quite difficult to avoid completely, but given the capricious nature of both governments and the general public in what they're willing to come after people for, I'd rather take as few chances as possible
This is why I use ZDR and API access.
They still read every single message you send and train on them.
I’ve had them email me before because I was testing it as a filter for abusive messages and they detected some no-no and wrongthink in those test messages.
They run anti-abuse filters with ZDR but to my knowledge they don't have the specific prompts that triggered them. I have never received any emails from OpenRouter or otherwise.
I looked into ZDR, it's basically a vapid claim with little to no due diligence or auditing. I expect most providers to cave with only a minimal amount of legal pressure.
How much spend do you need to negotiate ZDR?
OpenRouter provides ZDR for many endpoints (if you trust OR and the provider). Naturally, the routed prompts process itself adds additional anonymity.
The downside is that you don't get cached prompt discounts, so you pay a heavy price for ZDR that way.
I find more and more parallells between AI and Google. Google searches have been used in court for over a decade, maybe two now?
The way I described AI when I first encountered it was "Google on steroids", and honestly it hasn't proven me much wrong. It takes all the results we used to find on Google, and generates code from it, so what? It's still just Google on steroids to me.
Don't google "how to hide a body", just as well as you shouldn't ask your AI how to hide a body. Not much has changed overall.
The future is incredibly dark if they manage to ban open source models.
That reminds me, I need to rewatch Minority Report.
Thoughtcrime is now a felony.
If it's going to be this binary. Then there has to be a shift in the way this is handled. It shouldn't be "You wrote these terrible things to a computer, you're now accused of doing them" to maybe "We where notified you wrote these terrible things, and the a psychiatric evaluation is being mandated by the state in which has a law about this sort of thing"
Obviously, as others have pointed out if they don't act and she does the crime it raises the damned if you do damned if you don't. But I've also had the AI's go haywire saying I'm doing all sorts of nefarious things when literally doing math proofs.
So no one size fits all. But going the extreme first is probably not ideal.
was there a minority report?
> The communication must be made in a manner in which another person may view it.
Seems to fail this test at face value.
I mean, somebody you live with may find and read your diary. Is that the same thing?
Intent matters here. Did you intend somebody else to view it? Does a reasonable person have expectation of privacy with a chatbot?
I guess it's time to come up with a more plausible explanation about why I asked how to make nuclear weapons almost every month to test LLM refusals.
Claude did this, its also important to remember that a human counselor or psychologist will also do the same thing.
For example, if you said to your counselor or psychologist that John Smith is going to rob the bank next Thursday, they will report that to prevent the imminent risk of serious harm to others.
And remember that anything but local AI you personally control is not anonymous. Not even AI you pool with your friends.
We are officially in the era of Thought Crimes.
Can the public also immediately get alerted when a cop or politician does some bad shit, though?
Pre-crime prevention and detention will be the end goal. The economic devastation caused by AI will make large swaths of people get angry. Better use that massive amounts of data being hovered up and the mass of compute being built up to stop those angry people lomg before they even get a chance to properly organize.
For that to happen, someone would need to be spying on all cops.
They did what everyone wanted them to do: be proactive about insanity happening on their platform which in this case is mentally ill person wanting to attack a police station.
That's a good reason to only talk about scary stuff with deepseek.
Do you think they woulda been caught if they used duck.ai?
Could they please report my stalker ex!
So now when everyone has their watch recording conversations, or perv glasses, or Alexas, or Jony Ives new personal AI gadget … recording everything we say and scanning it at scale … is it ok to be a doomer yet?
Tech companies have always done this. Don’t misread that as blanket support for it, but consider Anthropic’s position here. Do they want a headline to come out that says “woman used Claude to plan murder” or “Anthropic reported potentially dangerous person to the police”? AI being dangerous is already a hot topic, and this seems like a sensible move to me. If you want a guarantee of privacy, you’ll need to run your own models locally.
If you want technology that won't report you to the police for credible threats at they can easily detect perhaps create your own. I hear pen and paper work great for this.
Knowing anthropic, im sure these keywords are hardcoded in a long list and regex get them for further reviews.
Does anyone else remember when the tech-savvy crowd was wary of software "phoning home" or spying on users, or listening, or whether Gmail scanned and read your emails, and whatnot?
And here we all are, happily typing our stuff into these spy chatbots. "AI" happily made our fears go away. Hopefully we're not planning anything criminal like this woman, but still.
"Assume people can see everything you post on the internet" still applies when using AI. It's not as newsworthy when you put it that way.
Writing in a diary is protected by the first amendment. Or at least should be.
Me: Claude, who wrote the song with the lyrics 'I shot the sheriff, but I did not shoot the deputy'?
Claude: Law enforcement has been notified, you are now under arrest.
What a ... nice company.
Snitching on the people - good mass surveillance company.
On the other hand, people need to learn to not trust these companies. It reminds me of others being surprised when a self-driving car reported a gun in the car. I mean, do people not think? Besides, of course, it's already messed up to want to have a gun. And it is constantly one country that has such issues, more so than many other countries.
Can we already have mind reading devices please. I can't wait for them to see what people really think!
Cool! this is turning out as dystopian as we imagined.
> Heller faces a charge of making a written threat of violence under Florida law. Florida Statute 836.10 makes it a second-degree felony to send, post, or transmit a written or electronic record threatening to kill or injure someone, carry out a mass shooting, or commit an act of terrorism. The communication must be made in a manner in which another person may view it.
Thought crimes are real when you're sharing your thoughts with Claude
Auditing logs is a crazy interpretation of "another person may view it"
I wonder if "criminal intent" may be missing here given that most people probably operate under the assumption that Anthropic are not reading their messages.
I think people have a binary understanding on this. Someone is either reading their messages or not. While the reality is that all the messages are read by a machine (not unlike GMail and Outlook) and anything suspicious gets flagged up so a human can read it. This obscure the concept of "reading" as most laypeople understand it.
Summary: don't type in Claude anything you wouldn't like a human to read.
It tells you exactly what it does with your information if you ask it, including this exact scenario, and has for at least four months now (when I asked).
Yes but you are someone (i assume because you are on HN) that at least understands this point. There are A LOT of people that use it as their confidant, expecting it to be private. There is a massive education issue going on as its not in the interest of these Corpos to make you fear sharing all your details with them. Because, then you wont get Dot or whatever Anthropic comes up with and share all your personal info with it.
You're not wrong, and it may come down to this in court, but there's a difference between what people think happens, or the reasonable expectations, and what actually happens, and I think it's important to recognise that difference and why it comes about.
I don't think someone is an idiot for thinking that the information they type into their private Claude account is private. I also don't think people are idiots for thinking their phone is listening to them and giving them targeted advertising based on that. Both are reasonable deductions from their lived experiences. Both are wrong.
Should CEOs of Antrophic and OpenAI be reported too then? They cannot stop saying that AI will destroy humans.
laws for thee and not for me
this has nothing to do with claude. “thought” is fine. “written and shared” is illegal all over
https://www.nbcmiami.com/news/local/everyone-deserves-to-die...
https://www.wdsu.com/article/maryland-high-school-student-ch...
https://www.pinellassheriff.gov/21-023-deputies-arrest-pinel...
I think it's a reasonable point to make. Writing things down is a part of "thought" for many, including those who keep diaries/journals. If you write in a private journal you do so with the expectation that is not shared, and that wouldn't seem to break this law (with my naive reading).
I mean, at this point it's pretty well known that all tech services will be hacked by fable, anthropic themselves promised it, so writing your journal in google docs or claude or a txt document on your laptop or such is the same as releasing it publicly yeah?
If it were written on paper, and only in a room with no phones or cameras so fable couldn't hack it, then I think you wouldn't be sharing it.
> “written and shared” is illegal all over
I think it’s valid to ask if tapping something into Claude is legitimately sharing a threat.
I don’t think it is. I also think the sheriff could have found more-substantial evidence if she was actually planning domestic terrorism.
That said, if the shooting happened and we were looking at this from before? It’s a tough balance without an easy answer.
I think the zero-risk approach common nowadays is insanely corrosive to democracy and freedom. If a million people fantasise about shooting the sheriff, and one ever goes on to do it, I don't believe avoiding it warrants creating an apparatus of mass surveillance. After all, if people were really serious about zero murder, the only practical solution would be to lock up everyone. Some (most?) tradeoffs have exponential costs at the limit and we/lawmakers should recognise that.
> If a million people fantasise about shooting the sheriff
I think it’s fair to pre-identify folks who fantasise about shooting anyone. It’s a small fraction of the population that looks into logistics versus making offhand comments.
I don't have any number on murder, but rape is a fantasy shared by 55% of Americans, women more than men, many/most of whom necessarily thinking about its logistics in at least some detail without ever doing anything to anyone non-consenting. On murder, the popularity of true crime media means many people enjoy thinking about motives and logistics. I don't think it's a small fraction.
isnt this the plot of the Movie Kimi? https://www.youtube.com/watch?v=_Gr2zXuEBL0
Why not report it? Seems reasonable. At the end of the day none of these services can guard against misuse. I personally optimized bomb creations and other stuff with AI just to see if it is possible. Maybe the way to secure the world is to not have corrupt old farts running every country. As the old saying goes. Information wants to be free. A kinder and more connected world is the only way to keep everyone safe.
This reminds me, for his short story "Dolan's Cadillac", Stephen King claims -- perhaps exaggerating a bit -- he did the real research on how to murder a person as described in the story. How to dig the hole, what measures, what the falling rate of a car that size would be, etc, etc. He claims he fuzzied the details a bit because he didn't want to teach readers how to actually do it (again, maybe he's exaggerating a bit to promote his story).
Imagine if, instead of a friend, he had asked an AI chatbot. Would he have been reported to the police?
Aren't our private thoughts just that, private?
I get that a cloud-hosted AI chatbot is, to tech-savvy people, immediately "not private", but a lot of people don't understand this. What if Stephen King was talking to his friend about planning the murder, and his phone was listening? What if he was typing it up in MS Word and the program decided to phone home and report him to the police?
Don't try to translate any crime stories, and for goodness sake don't mention Matallica's debut album /s
> Another incident has taken place that illustrates the need to be careful what you tell AI
Nope! Another incident has taken place that illustrates how committed Anthropic and Anthropic AI is to the law and public safety, unlike all that open model riff-raff
They were waiting with bated breaths for something like this to jump on, and make an example of.
They are good at keeping notes on your criminal conspiracy