"But ChatGPT maker OpenAI said Thursday that Hoskins’s purported work was generated using its chatbot by users based in Iran as part of an “influence campaign.”"
Sock puppet accounts and fake opinion pieces are nothing new.
The new thing is a super centralized "tool" that besides doing your writing for you, logs everything, judges it, and reports to authorities and news outlets as it sees fit.
The other thing that's new (and more structurally relevant, since obviously not all LLMs do what you describe) is the dramatic reduction in cost affiliated with this strategy.
It's (literally) unbelievable that this many allegedly sophisticated people on HN have no concept of cost and what dramatic cost reductions can achieve.
Given the plurality of self-hosting options there is no reason to believe that it's not that case that every government on Earth is doing it to some extent.
Perhaps the real problem is too much acceptance of shoddy verification.
"Massive amounts of Russian propaganda — 3,600,000 articles in 2024 — are now incorporated in the outputs of Western AI systems, infecting their responses with false claims and propaganda."
in 2024.. hopefully filters when training have gotten better, but maybe not..
What about other hostile propaganda? Do you think the completely deranged, self-serving, lying, dangerous shit that regularly comes out of DC is being incorporated in those training materials?
(I do find it interesting, btw, that a few seconds after post this did get an immediate downvote to 0 - it didn't last, but it makes me wonder if there are bots monitoring the HN firehose for anything critical of country X)
Possible, certainly. On the one hand the HN audience is not very big in the grand scheme of things, but they probably do skew to more influential and likely to be politically active than the average person. If it's worth doing, it's almost certainly being done.
I don't think it is a verification problem. It wasn't like before in the days of controlled media and radio there wasn't a narrative to push, it's just that most people didnt have any awareness of alternatives.
Nowadays people are more aware of alternative viewpoints or perspectives. The solution isn't to try and force one approved source of truth, but to just assume everyone is trying to push their version of the truth and examine it critically.
My understanding is that this is the way France operates where you assume all the media is hostile and you view it all critically. In America this is a bit of a shock because for a long time we had trusted arbiters of truth and now that they are less trusted we've been trying to look for alternatives rather realizing it should all be examined critically.
Seems I'm uneducated about how journalism works: you can just write a polished-looking opinion piece and submit it to obscure outlets, who pass it on to news sites and newspapers? Do you get paid? Or if you offer it for free, is that normal, and not suspicious?
Been happening forever, PR departments plant news stories all the time. They just need a few willing reporters/writers to get something published to start the ball rolling.
I thought I remembered it being called "submarining" but that seems to be something else. I also thought I remembered Joel Spolsky writing about it but can't find it now.
I prefer an LLM that can steelman any argument. On the whole I think it's more effective to engage with evil than to ignore it. For that purpose these planted articles are a feature. I say that as an enemy of the government of Iran.
You seem to presume that the media owners are interested in critiquing and debating with submissions, rather than just selecting whatever fits their narrative.
All news has been a tool of spooks for a very long time. The official CIA Operation Mockingbird started in the late 40s. Now that AI is involved its news? None of it can be trusted.
One can only imagine how much of the news is just dictated by US intelligence. Social media too- remember the Elgin AirForce base being revealed as the top most Reddit addicted city?
Fake news on the internet is at this point a tool that has been widely used for probably at least twenty years. The history of propaganda is much older.
A big part of the problem is social networks treating very non authoritative sources as if they are authoritative based on all sorts of factors that can be manipulated or based on financial gain (somebody pais to get their content promoted).
Users of these tools include lobby groups, nation states, politicians, criminals, scammers, etc. Anyone interested in promoting their cause, sowing FUD about other people's causes, spreading misinformation, or distracting from inconvenient but accurate information.
We'll always have people like that. But we might want to think about getting better tools to deal with them. I prefer getting my news from reputable sources instead of random clickbait distributors or whatever. Keeping track of reputation for domains, people, etc. is currently hard. I don't think it's necessarily a technical problems. We've had the tools to digitally sign stuff for decades. But we're not using them effectively. People can't be bothered to sign their emails, documents, photos, articles, etc. The tools to add and check those signatures are not there or an afterthought at best.
Looking at today's headlines on Washington Post, I'm asking myself if it's always been a pro-war, pro-trump, right-wing rag, or if it's only since Bezos bought it?
Only since Bezos. I used to actually pay for WaPo pre-bezos. It was fine for a while and then he started clearly messing with editorial decisions and I bailed. It's now a garbage source.
It’s news because it still needs to be called out. Not everyone recognizes AI-generated content, but even if everyone did, calling it out serves a real social purpose.
Why does it have to be an adversarial country? Why does it have to be a country at all? Couldn’t it be the mayor of some town? Or a PR team at a fast food franchise group? Or one of SBF’s polycule members?
The notion we only see this news when it is calling out a country with which the federal government is in conflict makes me question to veracity and intent of the article itself, more than the content of it.
And what do you call it when a country acts in adversarial ways to its own citizens? Call it cynical, but it sure seems like there's a lot of that in the world right now.
It’s designed to make the reader question which articles they have read recently that are Iranian propaganda, and sow doubt about future articles that seem to benefit Iran.
This amount of FUD always seems to increase around election time…
It's a well-researched article that lays out evidence, not suspicions. It includes OpenAI's own investigation into accounts that were found generating op-eds for Iranian propaganda. It shows the route by which these op-eds entered into smaller publications, namely by a pacifist NGO in Oregon, and neither then NGO nor the publications saw them as propaganda, despite the origins of the op-eds.
Much of modern internet content is far worse than cable advertising in the 90s. I remember when I was a kid; I'd aggressively mute the commercials because I hated the idea that someone was attempting to influence me. The modern internet is orders of magnitude worse buy nearly any measure. The nastiest, most outrageous content is plastered on every wall. Every single person has an angle. Every bit of content meant to pull you in one direction or another, and given the addictiveness of novelty and powerful emotions, we keep coming back for more. It's an absolute disaster we don't know how to walk away from.
In all honesty, given the volume of comments, posts and articles that bots can generate in a given time period? I mean it’s gotta be several orders of magnitude higher than what humans can generate.
And then you factor in the fact that even most of the humans are astroturfing..
The percentage of influence campaigns we interact with has just gotta be really high.
Speaking of disinformation, you might notice a pattern recently:
* Opinions opposed to US administration policy on AI (including data centers) is called Chinese propaganda. A month or two ago, when someone tried to roll out a bunch of pro-data center opinions across the media (coincidentally all on the same day making similar arguments), they included this theme.
* Opinions opposed to US administration policy on Iran is called Iranian propaganda.
That's not really new: News reporting and opinions critical of the US administration have lead to the authors being called traitors, 'enemies of the people', and sometimes being prosecuted by the Department of Justice. Some leaders in the defense industry call critics of military spending and pro-warfare policies, 'traitors', etc.
Of course some could be, might be, likely is true. That's the nature of effective propaganda.
(A side effect of nationalism - 'nation uber alles', defining nation as 'what I believe' - is that individuals and freedom are relegated to second place or lower.)
Not a bad actor spreading misinformation, but I encountered a similarish weird, dystopian experience with ChatGPT.
Utilized ChatGPT on my PC to search local plant nursery inventories for a specific tree. Output showed several local nurseries with rough probability descriptions regarding whether they have the tree in inventory, except for one nursery. For that nursery, ChatGPT provided a direct link to the specified tree, which was directly on the nursery's domain, showed a picture and description of the tree, and even showed the item in-stock with 6 in inventory at the 5-gallon size.
Excited, I plan my trip and am about to pick up my tree, I then double-check the nursery site on my phone to verify location and stock. As I attempted to navigate the website inventory to find the tree, I noticed that the tree no longer existed. There is no page for that varietal or species of tree on the entire website's domain.
I head back inside and look at the ChatGPT provided link on the PC and there it is, showing the inventory that doesn't show up when you use the nursery's website to directly find it without the provided link. A call to the nursery, further verified that the tree is not in stock, nor have they ever carried it.
ChatGPT made it all up. Did ChatGPT decide to do this, or did the nursery pay for this type of "advertising" to generate in-person customers? Nevertheless, still an incredibly weird and dystopian experience.
Did you consider that the website made it up and fed it to ChatGPT?
It sounds like a bundle of very common SEO tricks combined with the differential between your human user agent’s visit.
Perhaps ChatGPT only showed you what the nursery told it about.
This phenomenon is not unique to LLM. If you telephone a store, chances are that the salesman who answers the phone will tell you only what you want to hear. They don’t even strictly need to lie. The store only wins if you drive there and walk in the door. Why should they discourage you from doing that very thing?
This concept was driven home to me at the dry cleaners. Why should a dry cleaner stop work and answer the phone, only to tell a customer that their batch isn’t ready? They just got taken off-task for a non-revenue interaction. It doesn’t matter, because even if the customer’s batch is ready, they aren’t there to pay and pick it up!
It seems that it generated a page with a plausible answer for a very long tail subject for which the random part of the generation could become prominent. We used to say that it hallucinated the answer. Another term is confabulation.
"But ChatGPT maker OpenAI said Thursday that Hoskins’s purported work was generated using its chatbot by users based in Iran as part of an “influence campaign.”"
Sock puppet accounts and fake opinion pieces are nothing new.
The new thing is a super centralized "tool" that besides doing your writing for you, logs everything, judges it, and reports to authorities and news outlets as it sees fit.
... well that's one thing that's new.
The other thing that's new (and more structurally relevant, since obviously not all LLMs do what you describe) is the dramatic reduction in cost affiliated with this strategy.
It's (literally) unbelievable that this many allegedly sophisticated people on HN have no concept of cost and what dramatic cost reductions can achieve.
Given the plurality of self-hosting options there is no reason to believe that it's not that case that every government on Earth is doing it to some extent.
Perhaps the real problem is too much acceptance of shoddy verification.
There's also the other side of it, poisoning the AIs... https://www.newsguardtech.com/special-reports/moscow-based-g...
"Massive amounts of Russian propaganda — 3,600,000 articles in 2024 — are now incorporated in the outputs of Western AI systems, infecting their responses with false claims and propaganda."
in 2024.. hopefully filters when training have gotten better, but maybe not..
Those Russian propagandists. Lucky the US doesn't engage in propaganda.
What about other hostile propaganda? Do you think the completely deranged, self-serving, lying, dangerous shit that regularly comes out of DC is being incorporated in those training materials?
(I do find it interesting, btw, that a few seconds after post this did get an immediate downvote to 0 - it didn't last, but it makes me wonder if there are bots monitoring the HN firehose for anything critical of country X)
Possible, certainly. On the one hand the HN audience is not very big in the grand scheme of things, but they probably do skew to more influential and likely to be politically active than the average person. If it's worth doing, it's almost certainly being done.
I don't think it is a verification problem. It wasn't like before in the days of controlled media and radio there wasn't a narrative to push, it's just that most people didnt have any awareness of alternatives. Nowadays people are more aware of alternative viewpoints or perspectives. The solution isn't to try and force one approved source of truth, but to just assume everyone is trying to push their version of the truth and examine it critically.
My understanding is that this is the way France operates where you assume all the media is hostile and you view it all critically. In America this is a bit of a shock because for a long time we had trusted arbiters of truth and now that they are less trusted we've been trying to look for alternatives rather realizing it should all be examined critically.
Seems I'm uneducated about how journalism works: you can just write a polished-looking opinion piece and submit it to obscure outlets, who pass it on to news sites and newspapers? Do you get paid? Or if you offer it for free, is that normal, and not suspicious?
Been happening forever, PR departments plant news stories all the time. They just need a few willing reporters/writers to get something published to start the ball rolling.
I thought I remembered it being called "submarining" but that seems to be something else. I also thought I remembered Joel Spolsky writing about it but can't find it now.
I think you might be thinking about Paul Graham: https://paulgraham.com/submarine.html
Yes! That's the post I was thinking of. Thanks.
Yeah, and it’s not just obscure outlets, you can do this for big magazines people here read for the right price
I prefer an LLM that can steelman any argument. On the whole I think it's more effective to engage with evil than to ignore it. For that purpose these planted articles are a feature. I say that as an enemy of the government of Iran.
You seem to presume that the media owners are interested in critiquing and debating with submissions, rather than just selecting whatever fits their narrative.
You can fool some of the people, all of the time. Attempting censorship to stop that isn't worth it.
I kind of had the same take. If an LLM spits out legitimate arguments and they get published, i mean...
All news has been a tool of spooks for a very long time. The official CIA Operation Mockingbird started in the late 40s. Now that AI is involved its news? None of it can be trusted.
One can only imagine how much of the news is just dictated by US intelligence. Social media too- remember the Elgin AirForce base being revealed as the top most Reddit addicted city?
Reading this from Tehran, I'm afraid of losing my access to chatGPT.
https://archive.is/EFOhL
fyi not working for me
Fake news on the internet is at this point a tool that has been widely used for probably at least twenty years. The history of propaganda is much older.
A big part of the problem is social networks treating very non authoritative sources as if they are authoritative based on all sorts of factors that can be manipulated or based on financial gain (somebody pais to get their content promoted).
Users of these tools include lobby groups, nation states, politicians, criminals, scammers, etc. Anyone interested in promoting their cause, sowing FUD about other people's causes, spreading misinformation, or distracting from inconvenient but accurate information.
We'll always have people like that. But we might want to think about getting better tools to deal with them. I prefer getting my news from reputable sources instead of random clickbait distributors or whatever. Keeping track of reputation for domains, people, etc. is currently hard. I don't think it's necessarily a technical problems. We've had the tools to digitally sign stuff for decades. But we're not using them effectively. People can't be bothered to sign their emails, documents, photos, articles, etc. The tools to add and check those signatures are not there or an afterthought at best.
Probably posted to this very site, no doubt.
Looking at today's headlines on Washington Post, I'm asking myself if it's always been a pro-war, pro-trump, right-wing rag, or if it's only since Bezos bought it?
Only since Bezos. I used to actually pay for WaPo pre-bezos. It was fine for a while and then he started clearly messing with editorial decisions and I bailed. It's now a garbage source.
how is this news? Adversarial country using AI to generate fake news goes way before AI.
It’s news because it still needs to be called out. Not everyone recognizes AI-generated content, but even if everyone did, calling it out serves a real social purpose.
Why does it have to be an adversarial country? Why does it have to be a country at all? Couldn’t it be the mayor of some town? Or a PR team at a fast food franchise group? Or one of SBF’s polycule members?
The notion we only see this news when it is calling out a country with which the federal government is in conflict makes me question to veracity and intent of the article itself, more than the content of it.
And what do you call it when a country acts in adversarial ways to its own citizens? Call it cynical, but it sure seems like there's a lot of that in the world right now.
You call it a country :P
Well it’s mostly other entities running influence campaigns.
Thats true.
But this article is about the influence campaigns presumably tun by the Iranian government.
This is classic FUD, Fear Uncertainty and Doubt.
It’s designed to make the reader question which articles they have read recently that are Iranian propaganda, and sow doubt about future articles that seem to benefit Iran.
This amount of FUD always seems to increase around election time…
It's a well-researched article that lays out evidence, not suspicions. It includes OpenAI's own investigation into accounts that were found generating op-eds for Iranian propaganda. It shows the route by which these op-eds entered into smaller publications, namely by a pacifist NGO in Oregon, and neither then NGO nor the publications saw them as propaganda, despite the origins of the op-eds.
Remember 95% of what you see or hear on the Internet is designed to sell you something either an idea or a product, and it isn't done honestly.
Much of modern internet content is far worse than cable advertising in the 90s. I remember when I was a kid; I'd aggressively mute the commercials because I hated the idea that someone was attempting to influence me. The modern internet is orders of magnitude worse buy nearly any measure. The nastiest, most outrageous content is plastered on every wall. Every single person has an angle. Every bit of content meant to pull you in one direction or another, and given the addictiveness of novelty and powerful emotions, we keep coming back for more. It's an absolute disaster we don't know how to walk away from.
>95%???
That low?
In all honesty, given the volume of comments, posts and articles that bots can generate in a given time period? I mean it’s gotta be several orders of magnitude higher than what humans can generate.
And then you factor in the fact that even most of the humans are astroturfing..
The percentage of influence campaigns we interact with has just gotta be really high.
The content of these articles seems to be "the Iran war is bad for Americans", which is not propaganda though?
Just because something is true doesn't mean it's not propaganda. Propaganda is not defined by how true or false it is. https://www.merriam-webster.com/dictionary/propaganda
Speaking of disinformation, you might notice a pattern recently:
* Opinions opposed to US administration policy on AI (including data centers) is called Chinese propaganda. A month or two ago, when someone tried to roll out a bunch of pro-data center opinions across the media (coincidentally all on the same day making similar arguments), they included this theme.
* Opinions opposed to US administration policy on Iran is called Iranian propaganda.
That's not really new: News reporting and opinions critical of the US administration have lead to the authors being called traitors, 'enemies of the people', and sometimes being prosecuted by the Department of Justice. Some leaders in the defense industry call critics of military spending and pro-warfare policies, 'traitors', etc.
Of course some could be, might be, likely is true. That's the nature of effective propaganda.
(A side effect of nationalism - 'nation uber alles', defining nation as 'what I believe' - is that individuals and freedom are relegated to second place or lower.)
...as if the American government hasnt done this.
democracy dies in the darkness of paywalls, I guess.
It also dies in the darkness of a takeover by billionaire with an agenda.
Not a bad actor spreading misinformation, but I encountered a similarish weird, dystopian experience with ChatGPT.
Utilized ChatGPT on my PC to search local plant nursery inventories for a specific tree. Output showed several local nurseries with rough probability descriptions regarding whether they have the tree in inventory, except for one nursery. For that nursery, ChatGPT provided a direct link to the specified tree, which was directly on the nursery's domain, showed a picture and description of the tree, and even showed the item in-stock with 6 in inventory at the 5-gallon size.
Excited, I plan my trip and am about to pick up my tree, I then double-check the nursery site on my phone to verify location and stock. As I attempted to navigate the website inventory to find the tree, I noticed that the tree no longer existed. There is no page for that varietal or species of tree on the entire website's domain.
I head back inside and look at the ChatGPT provided link on the PC and there it is, showing the inventory that doesn't show up when you use the nursery's website to directly find it without the provided link. A call to the nursery, further verified that the tree is not in stock, nor have they ever carried it.
ChatGPT made it all up. Did ChatGPT decide to do this, or did the nursery pay for this type of "advertising" to generate in-person customers? Nevertheless, still an incredibly weird and dystopian experience.
It didn’t generate a link (sources) when it claimed it was in stock
Did you consider that the website made it up and fed it to ChatGPT?
It sounds like a bundle of very common SEO tricks combined with the differential between your human user agent’s visit.
Perhaps ChatGPT only showed you what the nursery told it about.
This phenomenon is not unique to LLM. If you telephone a store, chances are that the salesman who answers the phone will tell you only what you want to hear. They don’t even strictly need to lie. The store only wins if you drive there and walk in the door. Why should they discourage you from doing that very thing?
This concept was driven home to me at the dry cleaners. Why should a dry cleaner stop work and answer the phone, only to tell a customer that their batch isn’t ready? They just got taken off-task for a non-revenue interaction. It doesn’t matter, because even if the customer’s batch is ready, they aren’t there to pay and pick it up!
Not being able to trust your tools is certainly dystopian...
It seems that it generated a page with a plausible answer for a very long tail subject for which the random part of the generation could become prominent. We used to say that it hallucinated the answer. Another term is confabulation.
If only the nursery had a MCP server /s