If the people in charge of the AI companies understand the risk I don't know, but I think it's a good point that they mainly care about the money. These are also people who aren't accustom to cleaning up their own mess, so what do they care if something goes wrong.
The carelessness is also clear in how AI is rolled out. It's being presented as a disruptive technology, as is everything else presented by modern tech companies. The goal is a very large payout in a very short period of time, it's not to make anything better, or worse, either is just a side effect.
Had LLMs first been presented as a "better IntelliSense", where rather than auto-completing function names, your editor can now suggest implementations, very few people would have been upset. Same for writing, it's just a better version of Grammarly, it's fine, or as a tool in drawing software. The issue would have been, for the AI bosses, that this takes time and doesn't create trillion dollar companies. It only creates companies that they'd actually have to run, long term, and that's boring.
> Had LLMs first been presented as a "better IntelliSense", where rather than auto-completing function names, your editor can now suggest implementations
Aside from the AI dungeon app, this was my first real exposure to LLMs wrt github's copilot extension. It sucked, and did a lot of stuff wrong, but they got exactly 5 bucks from me to try it.
It's sorta crazy to think how many years ago that was now.
> The issue would have been, for the AI bosses, that this takes time and doesn't create trillion dollar companies. It only creates companies that they'd actually have to run, long term, and that's boring.
I feel like you’re not acknowledging the inevitability of the technology being developed further. You’re also ignoring competition and the basic fact that people do research and push the boundaries of their field naturally.
If the major frontier AI labs had decided, as you said, to make a fancy autocomplete and a better Grammarly and then simply stop and coast, do you really believe that every AI researcher and entrepreneur in the entire world would look at the situation and decide that the status quo is where it should end?
Humans don’t work that way. People would get ideas and want to push the limit. If the frontier labs stopped developing their technology, other companies would be created to carry on. Academic researchers would push the limits further. It would be discovered that the AI could go further and we’d get back to this same eventuality with a different set of people leading it.
I also disagree with your characterization of this as a get rich quick scheme for AI leaders who want to make a lot of money and then be done. I think this characterization feels truthy if your only conception of startups is as a gamble to retire early, but the people who build and run these companies actually do like building and running companies. If the AI leaders and executive management you’re talking about were separated from their jobs today, even with billions of equity, most of them would be out starting new companies next month because that’s what people like that do. I’m not saying this to “glaze” them or whatever I’ll surely be accused of in the comments, I’m stating some facts about this space. You don’t need to look far to find examples of this already happening.
I really don't like the excuses that are inherent to the "well it was inevitable" line of thinking. Oppenheimer was still responsible for the bomb despite it being "inevitable" in a similar way.
Despite this, when discussing the egregious behavior of the frontier labs, there is vanishingly little acceptance of the idea that 1) yes, it probably was inevitable in some ways AND 2) those involved, particularly at the top, are still responsible for the consequences of their (bad) decisions.
I am not sure what gripe you have with the labs, but I think if you are worried about AI then you got shockingly lucky with the lab leadership. My biggest gripe has been the campaign of negativity against ai.
I have also not been able to lean towards a side in inevitable v/s responsible.
1. Is it both inevitable and still they are responsible?
2. Or it was inevitable and they were just an instrument to make it happen.
3. Or it wasn't inevitable and if it were not for some specific people then it would not have happened?
I am leaning more towards 2. But if that is the case why are only specific people reaping the benefits of this?
If it is 3 or 1, then why aren't we holding them accountable?
"Inevitable" means "same technological progress would be achieved anyway, even if these people have made less damaging choices".
So yes, they are 100% responsible for damage they caused and did not had to happen. And no, "I had no choice, because it was the only way to become billionaire" is not an excuse.
They were NOT "instrument". No one manipulated them. No one forced them. Nothing forced them. They themselves wanted to and did not cared
Was it? We nearly destroyed ourselves with nuclear weapons several times in the last century. I don't think it was a good call just because we've gotten lucky so far.
Imagine a bag filled with marbles. We can't see inside the bag, we can just pull marbles out of the bag. Some marbles are white - they improve the world in some way. Wifi. Trains. Electricity. Some marbles are grey - they are useful but dangerous. Firearms. Nuclear weapons. Gene editing. And some marbles are black. If we pull a black marble out of the bag, we hurt or destroy ourselves.
Humanity's strategy so far has been to pull marbles out of the bag as fast as we can. And that's been working out, because most marbles are white. But we're insanely lucky we haven't had a nuclear war. The USSR's early warning systems picked up 5 incoming american missiles in 1983. They didn't fire back because one of the officers on shift didn't believe the reports (why only 5 missiles?), and he declared the reports were fake.[1] If a different person was on shift, there's a good chance nuclear war would have broken out because of a sensor glitch. And that's far from the only instance. The list is terrifying: [2]
And now we're improving AI capabilities as fast as we can, despite it being pretty clearly another grey ball. AI has a lot of amazing uses. But why are we rushing so much? Doesn't seem wise.
It's not about merely knowing how to build a "badthing". It's about knowing why not to.
With AI, the point is who uses it for what.
Think about it like a magic wand that makes your wishes come true. Would you really want everybody to have one? Who should and who shouldn't? Who gets to decide?
And this sort of blanket argument is how you get bad science on the order of Prison experiments to Nazi scientists. Yes, academic research that pushes boundaries can be a $badthing
You operate within the "logic" of capitalism, where no other value exists but the (short term) economic one.
AI indeed goes far beyond "autocomplete" and other nonsense. But the question is, whether it's a good idea to just let the "market" run wild with it.
Short-term capital interests don't align with the public good. AI greatly scales capability, but without anything to restrain it, that implies those non-aligned interests overwhelming society's defenses, outright destroying it in the process.
The idea of rich people being some kind of benevolent caretakers, who independently and dependably foresee and manage the consequences of their actions in everybody's best interest, is laughably wrong already.
With AI, that reality gap will swallow whatever is left.
> You operate within the "logic" of capitalism, where no other value exists but the (short term) economic one.
I operate within the logic of how the world actually works.
Which I find infinitely more useful for practical topics like the reality of AI development in a global economy.
Fantasizing about alternative realities where all the leaders of the world unite and halt technology development without any of them secretly defecting and developing it further isn’t very useful to discuss. These things aren’t going to happen.
The market is irrelevant. In the 28 centuries of recorded history, the human race has pursued power in it's various forms (political, military or economic), without regard to the "public good" (which is really, just a polite euphemism for the established order and those that benefit for it).
Without capitalism, you wouldn't be able to restrain AI adoption any more than the armored knights of the middle ages were able to restrain the adoption of the firearm.
> Had LLMs first been presented as a "better IntelliSense", where rather than auto-completing function names, your editor can now suggest implementations, very few people would have been upset.
I'm not rooting for AI companies or hustlers that are out to ride the wave and get rich.
However your claim is just hyperbole. I've used it in cross-cutting projects to extend protocol on desktop, iot device, which involves holding premise long enough to co-develop and test. It is also fantastic in debugging multi-threaded systems.
This is an interesting theme that I see a lot: Some people are firmly anchored to an idea of AI capabilities that is a year or more behind the frontier.
Some times it comes from people who try out ChatGPT free tier or the Google AI summary and think it represents the state of the art.
More recently I suspect it’s from people who build their opinion based on being contrarian and cynical: They dislike the AI people and distrust everything they say. If the AI people say it’s changing the world, then the cynical and contrarian belief is to assume that the technology is actually very bad and they’re just lying to make a lot of money and retire.
These takes were more understandable in the earlier phases of AI when it really was very limited, but it’s getting harder to understand how someone can be thinking of AI as just a slightly better autocomplete or slightly better Grammarly at the end of 2026.
> but it’s getting harder to understand how someone can be thinking of AI as just a slightly better autocomplete or slightly better Grammarly at the end of 2026.
That wasn't the point. My point was that if you had rolled LLMs out like that, as embedded technology in existing products. Slowly, over time, you'd have a lot less cynical people. I think the hostility towards AI product in large parts comes from the disruption and glancing over the negative parts. What we're doing now is the equivalent of putting millions of cars on roads which five years ago had never seem a car, only horses. We're doing so without any understanding of how that might affect society. Social media was a similar story. It was slower, yet we still didn't catch the majority of the negative consequences.
If it turns out that the people behind social media, and smart phones, don't care about e.g. peoples mental well being, then why would I trust that the people behind AI wouldn't behave the same way?
There are people whose livelihood depends on them not getting it. There are people whose worldviews depend on them not getting it. Denial is a part of grieving. There are also people who like to troll the ai maniacs because ai maniacs are easy to troll. It has been curious to me to see all this as well, smart people who seem oblivious feels strange.
That was written in 2015 before LLMs were much of a thing and has been fairly spot on so far. This is not just about LLMs any more than the industrial revolution was just about steam engines.
There are a huge number of people involved in AI outside of the major AI companies both in smaller AI companies and lots of independent developers. They aren't having much success bringing AI to the masses and this is because they don't care enough about money and they aren't selling their ideas aggressively enough. There is this odd tendency to undersell achievements because they weren't motivated by the Right Stuff.
The AI companies are as much being pushed along by investors as they are in the driving seat, but the push is in the direction of getting real products in front of people and experimenting with this miraculous tech everywhere it could possible work.
Ignoring the implementation details, if we treat the whole global technology world as one single AI system made up of AI companies, independent developers/researchers, and end users, what would be the training goal and evaluation?
An excellent comment. It's very true - there's a deep sickness here which causes people to flee from the inherent value (relatively modest) and the longer, more tedious work of building a sustainable company around it.
> there's a deep sickness here which causes people to flee from the inherent value (relatively modest)
Value is simply the mutual understanding that something should cost a certain amount, and the system we've built emphasizes the need for profit above all else. When a set of companies comes along with a new, impressive technology claiming that they can automate away the white collar workforce, they're trying to capture as much value as they can.
It's not a moral sickness, it's market actors behaving rationally. Believing that people should be willing to leave money on the table because it's the right thing to do is a dead end, the entire system needs a rethink.
> It's not a moral sickness, it's market actors behaving rationally. Believing that people should be willing to leave money on the table because it's the right thing to do is a dead end
I don't agree that "I am just a market actor behaving rationally" somehow absolves you of any moral duty
I'm not an ethicist or anything but I have studied quite a lot of ethical frameworks and I've never encountered one that I think would agree with this
> Believing that people should be willing to leave money on the table because it's the right thing to do is a dead end, the entire system needs a rethink.
I didn’t expect a full throated defense of mugging and burglary on HN this morning, but here we are.
I enthusiastically agree. I've been on this ride since GPT 2 and 3, I did start out looking at the technology in a neutral or positive light. I did not think the stewards of this technology would be so irresponsible, its consequences would rival the Stalinist purges.
No matter what practical benefits this technology might bring, if this runaway marketing campaign isn't stopped, in just a couple decades we'll see what the complete collapse of civil society looks like, a world where the institutions no longer see value in maintaining an educated democratic populous, and can relax back into the comfort of feudalism.
I’m sorry, but the AI workers are as much to blame.
Or have you forgotten that Sam Altman was ousted by individuals that shared your concerns, and it was the OpenAI employees that rebelled against the board and wanted Altman back?
That HN post was infamous and nearly took down this site.
The AI formula is out of the bag. The Manhattan Project is over, trinity has happened.
If it weren’t OAI and Anthropic, it would be others.
If it weren’t US companies, it would be others.
It is far too useful now to stop large scale deployment, it isn’t a matter of specific people or countries not being cautious or prudent enough. The entirety of the Earth is at issue now and there will always be people who cause the tragedy of the commons (if indeed AI turns out to be harmful in the long run, which is not remotely certain).
Look at what happened with fossil fuels. Even if the entire global west totally ceased use, there are many countries that simply will not stop because of local utility. AI is the exact same way.
There is something deeply rotten about the US mindset though, them being American is definitely a problem.
> Look at what happened with fossil fuels.
The west had a 100 years head start, on top of that these countries are producing everything you consume, don't start pointing fingers. Chinese current co2 per capita is the same as the US in 1900
I remember reading about the development of neural networks in a magazine from the 1980s.
Releasing artificial intelligence for the use of all of humanity is better than it being controlled by one or more shadow groups. That's what the Americans did, and the Chinese followed in.
Had AI been invented in the 20th century, it would have been kept secret and used only by state actors to facilitate genocides at mass scales. But we live in different times.
Widespread mass realtime facial recognition usage is allowing governments to target and repress subsets of the population with an unprecedented scale and efficiency. ICE are flying facial recognition drones over sidewalks in downtown LA. Dissdents are having their facial geometries entered into secret databases simply based on political opinions.
Any new technology throughout history has first been used for war and genocide.
How could modern AI have helped those groups you mention in their endeavour? Really fucking easy and effectively, and you don't need a lot of imagination to think of how. From surveillance and interrogation to pinpoint determination of ethnicity and much more. Not to mention translation between human languages.
> Releasing artificial intelligence for the use of all of humanity is better than it being controlled by one or more shadow groups. That's what the Americans did
Uh? In what universe are openai and anthropic not shadow groups?
Are you misunderstanding on purpose? OpenAI and Anthropic let the public use their AI models. Shadow groups would mean that people wouldn't even know about the existence of AI.
Cool snark. I said it's better that all humanity have access to use AI rather than just a few people. That's pretty nice, compared to the alternative.
But if you're an angry hacker in full on culture war witch hunter mode, then of course nobody can say something that's even slightly different than full-on hate without you interpreting that as aligning with the evil witch.
Yes, the cat is out of the bag as to how to create this type of AI.
But that doesn't mean that everyone has to use it.
We're only starting to see the beginning of the negative impact of AI to the job market and society. If if turns out that allowing most jobs to be replaced by AI is bad for society and the economy (super high unemployment - nobody buying more then necessities), then wouldn't it be smarter to not do that?
If that's the case, then each country can decide for themselves.
Dario's "Machines of loving grace" UBI/foodstamp "utopia" sure doesn't sound like a recipe for success.
Which job are worth doing and which ones you'd rather the AI do?
AI should enable things that were previously impossible because of time constraints.
If time is money and we were previously trading with money, we will get back to trading time. Manufacturing time for instance. Compute times.
For commodities, that might be automated to some extent, I am not sure. The friction in exchanges is where the money is made and the friction here may be at the interface with the real world still.
> If it weren’t OAI and Anthropic, it would be others.
Doing what? Rushing to make trillion dollar companies, rather then developing the tech and their companies in a sustainable way? I don't even know what in the original comment this would be a response to.
Rushing towards creating machines that can give you vast amounts of military, political or economic power without regard to it's effects on the old order.
The particular form of the pursuit of power practiced by OAI and Antrophic - the creation of trillion dollar companies, is one of the most benign in recorded history. It's so unnaturally benign, that I doubt it's going to survive for much longer.
"he warned it could trigger a large-scale incident this year or next, such as a temporary blackout of financial or power systems."
I know this sounds unusual, but I do think this wouldn't be the worst thing that could happen. Sure, it would cause a bit of chaos, but it might also make us think twice about continuing down our current path of unfettered development, and possibly even make us much more cautious, which might in turn even help us with the climate crisis.
Because human beings in a large mass only respond to one thing: immediate difficulty. They don't respond to delayed difficulty like "the climate is getting worse" or "technology will make life more difficult in the distant future". They only respond to immediate pain.
So I think it would be a good thing if the financial system had a serious threat. Then people with money, who tend to offset future problems with technical solutions and financial padding, would start to become much more suspicious of technology and defund AI and wasteful things that are likely to threaten their safety.
I agree with your premise, but not your conclusion. The problem is not the scope or scale of things, but treating them in an absurdly stupid fashion. Why are power systems internet connected? It certainly enables some greater degree of power and flexibility, but it exposes them to potentially catastrophic outcomes in case of malfeasance. So what happened at some point is somebody convinced somebody else that the security was super duper ultra good, and that's simply not true.
And I'd go one step further to say that it probably can not be true. This has always been true and probably will always be true. All LLMs are doing is demonstrating that. And the LLMs of tomorrow will demonstrate that the most LLM hardened system of today is still just a vulnerable mess. So the conclusion is simple - things should not be connected to the internet unless the worst case scenario from hacking/exploiting is relatively low. If somebody wants to connect their coffee maker to the internet, for one can only imagine whatever reason, then the worst case scenario is a broken coffee pot - who cares, go for it. But connecting e.g. power, water, or other core infrastructure to the internet? No, never.
Just because something hasn't been proven to be a problem today does not mean it won't be a problem tomorrow. We shouldn't learned that yesterday, hopefully we can learn it today.
We are starting to get a taste. Credit Mutuel Arkea just admitted having been hacked. Over 100 millions gone.
Blockchain despite all its flaws, is the canary in the coal mine. They have been dealing with an increasing amount of cyberattacks. Because it all happens publicly in the blockchain, there is no way to hide it.
However major bank gets hacked in the tune of over 100 millions and we get to know it happened because somebody who has no relationship to the hack turned his eyes to a bunch of suspicious transactions happening on Ethereum.
Don't ignore the very real human cost this would bring. Every day the power is out, more vulnerable people would be harmed(including death) directly from not having electricity for essentials or from systemic failures like in food distribution and storage. Financial systems being down could also have extreme impacts on some people, but maybe easier to mitigate via human compassion.
Maybe the long term changes such blackouts would cause would be "worth it" in a utilitarian sense, but let's not hope for such damage.
You’re describing a weapon, and if AI managed to do that (I believe we are not far away), then this would probably increase the investment not decrease it.
Would it? I don't think so. Not if the weapon is an indiscriminate one and makes everyone worse off. Yes, some people (maybe military generals) might support it. But there would be near universal hatred for it, if people wake up one day and find their bank accounts not working or their eletricity not working.
This is why I think AI is so stupid. It's just too powerful for anyone to control, especially with the inertial mass of capitalistic society powering it through the incremental, short-term advantages it gives.
> Nuclear weapons are indiscriminate and make everyone worse off if used.
The difference is most people have not experienced any immediate effects of nuclear weapons. But people will experience the immediate effects of AI taking down the electricity grid, for instance.
Right, just like the war to end all wars totally prevented the nations involved from going to war with each other... for a grand total of 20 years before they decided to do it again. There are other more powerful and influential factors that decide the fate of nations. Even if you get your wish and everyone suffers and everyone hates AI then it will simply disappear into classified research projects and government run data centres. The NSA could easily absorb it but I'm guessing government contractors would also want a piece of the pie for weapons research.
If we're at the point where all that matters is scale and reinforcement training then the public facing aspects could entirely disappear and progress would still continue especially when given an infinite budget of taxpayer dollars.
"most people" are in the set of things the trillionaires don't give a shit about. Their market distortion has already made PCs unaffordable and electricity prices rise, among other adverse effects. What else has to happen so you take notice?
To add to your point, many people’s jobs had been affected by AI, job market data don’t lie. That didn’t stop the progress, it became exponentially faster.
I agree that people only really respond when something goes wrong badly enough that its impossible to ignore.
I disagree that something like this would be a good thing though. Whether it's a power outage, financial crisis, or something else people will die. If that's what happens and society ends up adjusting course after then that's what happens, but I can't call people dying a good thing.
This sort of thing was going to happen, with or without AI (state sponsored hackers, the US or China taking down systems they control, or whatever).
The later it happens, the more we will be down the path that lets it happen, and the worse the consequences. We are getting gradually worse consequences: AWS outages taking down websites, GCP outages taking down banks, attacks on fridges disrupting good supplies...
Its better for us to change path when something happens that causes dozens of people die rather than wait until its millions.
I don't think that's a good thing. But if we DON'T do anything, then even more poeple are likely to suffer. So while people dying is not a good thing, a thousand people dying is better than a million people dying because we don't do anything.
Same thing about burning your hand. It's not a good thing to burn your hand, but burning your hand once when you're young probably saves you from a more serious accident later on.
So Hinton doesn't trust the tech leaders but then the consequence of having governments censor LLMs is that only tech leaders and governments have unlimited access to AI.
If someone manages with the help of AI to hack into a vital system, then the system was already in a critically bad state beforehand. If developers can't use AI to check the code for security flaws, then security gets worse and not better.
I would posit that systems being hacked by AI were already vulnerable to the huge caches of secret zero days held by intelligence agencies and organized crime around the world. Which we know exist.
Insecure is insecure. These systems never were secure.
Of course, we are.
Back in the late 80’s early 90’s, no one working on ML was aspiring to develop systems which would be used to generate the kind of AI crap which plagues us today.The good is being smothered by the absolutely soul-crushing pointless.
AI exposes the Sturgeon's law (90% of everything is crap), and amplifies it much more, and anything good is drowned out. Most of human-generated popular music is slop. Most of human generated code is buggy, unmaintainable and full of debt. Social media and youtube have always been dominated by click-bait and mediocre time-wasting crap.
I hear you and i find it devastating, too. On top of that, the negative effects on the planet for having this technology vomit this soul-crushingly pointless crap is just depressing.
Before AI I was already being smothered by soul crushing pointless junk made by humans.
You’re placing blame on the wrong spot. It’s algorithms and social media incentive systems not AI. Our digital media incentives addictive minimum effort chum and slop. AI just makes it easier to make.
"We". Who "is" we? This isn't about you. or me. there was never a we. Before you open your mouth or put your pen down, wouldn't you think of the shareholder value for once?
My main fear is that the integrity of everything will be compromised, but everyone will keep acting as if it's not. I feel like it's already somewhat the case but I'm worried about it getting worse.
My prediction is that some people will be able to use AI to create essentially infinite money inside their bank accounts (through hacking); the prices of assets will grow to ridiculous sums as the people with infinite money buy up assets, but yields on all assets would soon approach 0%; still, one asset is going to grow at a faster rate than all other assets (E.g. an AI company backed by blockchain/tokenized shares that are impossible to hack) and everyone is going to start piling their infinite money into that asset; every other asset will crash because of the rapid, exponential price acceleration of the top asset making every other asset unattractive by comparison. Then fiat money would cease to exist and shares (or crypto tokens) of the top company will be used as money instead. Then prices of assets would stabilize again; denominated in the new currency... Then after that, we will finally be able to start innovating with AI.
I also think so. The weakness of current popular cryptocurrencies though, is the complexity and size of their codebases. Many above 100k lines of code. That's risky. Though the benefit is that they are somewhat heterogeneous (different nodes running slightly different code/versions) so it may be hard to find a vulnerability which affects more than 50% of nodes; and this is a hard requirement for compromising the integrity of that system.
Traditional finance has the benefit that they can more easily reverse malicious transactions, but if hacks become too common, it will be impossible to correct fast enough and in a reliable way. With traditional banking, a hacker only needs to hack a few nodes to compromise the integrity of the entire system.
So yeah I also think crypto will win in the long run. I would bet on a tokenized stock with AI attached.
I think the top asset will combine the AI which is best at hacking with the blockchain which is hardest to hack.
Ability to corrupt others whilst maintaining perfect integrity yourself.
“I think [Mark] Zuckerberg and [Elon] Musk and Sam Altman, they understand the risks, and they don’t give a shit. They’d rather have the money. That’s my belief.”
What's risk for people valued close to or over trillion dollars? Risk is meaningless when corporations get too big to fail.
Yeah, seriously. AI has been the single biggest technological improvement to my life in the last 20 years. If I could choose between smartphones and AI, I'll take AI any time of the day. Smartphones (in combination with social media) had a worse impact on society than AI has had and are not worth their advantages to me.
There's just something off about this mindset - why is achieving your life goals faster better? I think you haven't realized that life isn't a bunch of checklists you need to complete but rather about the journey, and when you do get that, you don't need to go fast.
Also the future where there is no sacrifice to achieve your lifegoals seems just unnatural to me.
The same applies to most of humanity’s other inventions, such as transportation, weapon, electricity... Ultimately, these technologies serve as instruments for rivalry, deciding the fate of the strong versus the weak.
As is the case with any technology in the hands of careless people, but unfortunately we can't wait for society to catch up morally to the technical standards, innovation is organic and breakthroughs have no timing.
> Hinton, 78, said that within a decade and possibly sooner, humanity will know whether it can manage the technology’s risks and enable what he called “radical abundance” or whether “we fail to manage the risks and … have a huge gap between rich and poor, huge cultural problems, massive unemployment, just society being torn apart”.
"It might turn out we were better off not opening Pandora's box" says man who helped open Pandora's box.
That's a touching sentiment or whatever but the nature of the thing is that it will inevitably be opened and that it cannot be closed thereafter. So (I think) we'd all be better off putting our efforts towards predicting the likely outcomes and figuring out how to make the best of them.
"Could" does a lot of work in these kinds of claims.
We need to build it as fast as possible because it could cure all disease. Ignore whether that's possible, how it might happen, whether all we're missing to do it is more intelligence, etc.
The only way I see us curing all disease is to throw out all the names we've given to collections of symptoms. What disease? I don't see no disease, you just have a sore throat, runny nose, and mild fever.
> We need to build it as fast as possible because it could cure all disease
Hell no. We need to risk pumping out even more CO2, causing more environmental disaster, ruining entire human communities, just to cure diseases? You might as well say it's better to be in prison because you're guaranteed three meals a day. The logic of some pro-AI people is baffling.
I have to spend extra time just trying to find AI-free things, AI-free writing, AI-free communities, and people who aren't addicted to AI. And get noticed/get a job by having to push through all this AI static. And having to interact with AI-based customer service agents that take 10x as long to resolve the problem as a human being. And having to wait a bit longer because there are 10x more 'Are You Human' Captchas slowing down website loading. And having to download OS updates that are 2x as large because they contain AI models. And having to double-check everything because there's so much more hallucinated crap out there.
The other day I hired an accountant and even he sent me an AI answer that sounded right but was actually bullshit.
In short, it takes a LOT more time just to find some sanity now.
I mean, it could if "we" "just" got out of globalized financial capitalism, changed everything about our political, economical, financial, social systems, etc.
Only 14 years old and accelerationists think we can just push the capitalism pedal further and somehow get social changes going their way
I don't think accelerationists think we can do anything. The thought process of the accelerationist is partly fueled by the 'excitement' of betting and playing musical chairs and hoping to be the first ones out holding a bag of money and partly fueld by the delusion that technological advancement is always good at any level of complexity.
To be honest, I think it depends on the environment. In the past, if you didn't come out of a good university, your access to knowledge was limited. But these days, in terms of access to knowledge, I think things have clearly improved, whether or not AI is illegally sharing knowledge.
Looking back now, was it really happier before AI existed? I'm not sure. In the past, when I wanted to find out whether I was right or wrong about some piece of knowledge or information, I'd contact several people and, far from getting an answer, I'd often be told not to bother them. But AI, in that respect, is the best teacher I can think of.
I think the times are continuously getting better. It's just that there's a problem of power concentration among the people who handle it. Still, if AI is a tool of that nature, then I suppose that can happen.
But considering what access to knowledge was like before AI came along, I think having AI is better.
I'm not sure of this. I remember back in the day finding people's blogs, reading old forum threads, participating in said forums - there was a ton of info out there and easily found. Now those kinds of sites are rare, and those that still exist aren't surfaced by search engines like they used to be.
AI has the benefit of having trained on all that material, it's a snapshot in a sense. But as people increasingly stop creating content like that and we turn to AI instead of asking for help on forums we stop growing the knowledgebase that AI draws from.
Knowledge was already extremely accessible in the age of the Internet, but it required time to read and learn. AI is making that same knowledge applicable without the prerequisite of learning.
Most people are not interesting in learning for learning's sake, they are motivated to learn something they can apply (see: kids complaining about how learning Math in school doesn't transfer to useful skills). Why learn about architecture and technical debt, if you can tell the machine to make it faster?
I don't think you're wrong, but fundamentally we're in an age of information overload, and this may be the new normal. In other words, the environment where you had to know things deeply and become an expert, like in the past, may be different from an environment where you know many things broadly and combine them.
I basically don't think you're wrong. However, books and reading have their information fixed at the point of publication, whereas depending on where you live or your community, the information you get buried in differs. Because I was in an Asian country, it was actually very hard for me to synchronize my programming skills with the latest technology in the US. The frameworks commonly used in my country are stuck at roughly 15 years ago. The time lag between US technology and my mainstream technology is about 5 years. In my area, it's about 15 years.
Anyway, thinking about your opinion, I think there are both valid points and dark points. I think learning for its own sake is important.
I agree that people should have an attitude of learning for its own sake, but I don't think it's mandatory. I think it should vary depending on the environment. Still... I do think having that attitude is a good thing. I like your idealism, senior.
However, that holds for people who are middle class or above and can survive. Because rich kids can picture the future and are given the right to love learning. But if you're poor, that's very hard. And when you grow up in such a poor family, there's an emphasis on it having to lead to practical skills.
And even if someone loves learning for its own sake, I think quite a lot of people love the honor that comes from learning. But that too can be called loving learning itself. It's purely a matter of individual values.
I don't think you're wrong, senior, but I also wonder whether the old ways are right in today's era of excessive information. It's just that because our lives and the environments surrounding us are different, we came to different conclusions. I upvoted because I agree with your thoughts. But there are points where I differ. Have a good day.
In East Asian countries, certification systems are still maintained, but in reality they're practically meaningless. If a certification system exists, people will take an interest in CS and work harder at it, or so you'd think—but in reality, the exam questions end up stuck in the old style. In other words, there's a big gap between the exams and the actual field.
Because a certification has to be fixed at a particular point in time. Korea's certification exams start with the 23 GoF patterns and go on to rote memorization of countless patterns and layered terminology.
The problem is that the way you learn for this certification is the old way, so someone who gets the certification ends up writing legacy code.
But if you always update the questions to the latest, then nobody can pass the certification. So, having seen how the certification system works in East Asia, I don't think it's all that good
It might turn out we were better off without nuclear weapons, but so far I’m happy we haven’t had another world war (in the traditional “total war” sense).
No way to tell until after, though. So far, mutually assured destruction is working. It works until it doesn’t.
Same with AI. Thus far we’re better off with it than without it. That may change.
I mean, look what we did with fossil fuels. Some people are even still willingly burning them, even though the data is clear that our use of them has already started the total destruction of our civilization.
Is there a name for this, a technology that is so insanely useful that there is no way we won’t exploit it, but where the worst case scenario is total collapse, and impossible to determine well in advance?
(Fossil fuels I suppose don’t count as even the energy companies knew unambiguously decades ago that large scale use would eventually bake the entire agricultural system out of existence.)
>so far I’m happy we haven’t had another world war
The reason for that was more that the 3-4 great empires involved in the previous two were exhausted, Japan was defeated, China was in no shape, and US vs USSR had no reason to push for one, since the Cold War era gave them plenty of breathing room, roughly half the world each.
Both the USSR and US fought multiple wars directly against the proxies of the other. They weren't simply hanging back and letting others do the dirty work. In most any other era one of those would have become a full blown war between the two. Both sides pushed the line of not directly involved and then both sides looked the other way versus escalating. Nukes made the alternative too much of a risk for both sides.
The scene is bleak, silent, desert-like with a few dead trees bleached white by unfiltered UV. Nothing moves. In the distance a cave, and from that cave
No it's a completely serious question. The goal has always been to replicate human/biological thinking processes in software, and now that this has been more or less achieved he's suddenly not happy about it. There's been plenty of opportunity to actually think about "what if all of this actually works" but they chose not to.
All economic activity emits CO2 directly or indirectly. The question is are the benefits worth the cost, and could the process be more efficient. A molecule of data center CO2 doesn't affect the climate any more than one that comes from producing cement, or getting a Labubu delivered overnight. If we want less CO2 emitted, it is best to treat emissions equally and let people decide what they value when they spend their money.
This is tangential encouraged by your point and other forays here. But one does wonder what arguments politicians (the power lies with them) understand when they make their pronouncements. The UK has for instance, prior to the new administration, totally and vociferously supported net zero. From the point of view (a fact for many but not for all informed authorities) that climate change is caused by CO2 and we are headed for a disaster - Net Zero is a logical step to trying to save humanity and the whole ecosystem. Now (November) I read that the UK powers that be, appear to have put it aside in actions at least. Countdown to Net Zero doesn't matter any more; open the oil fields, get rid of subsidies for photovoltaic and wind - and other measures. My point is not a political jibe but a stance I suspect to apply in other areas. Is climate change going to be a catastrophe or not? They should make up their minds.
If the people in charge of the AI companies understand the risk I don't know, but I think it's a good point that they mainly care about the money. These are also people who aren't accustom to cleaning up their own mess, so what do they care if something goes wrong.
The carelessness is also clear in how AI is rolled out. It's being presented as a disruptive technology, as is everything else presented by modern tech companies. The goal is a very large payout in a very short period of time, it's not to make anything better, or worse, either is just a side effect.
Had LLMs first been presented as a "better IntelliSense", where rather than auto-completing function names, your editor can now suggest implementations, very few people would have been upset. Same for writing, it's just a better version of Grammarly, it's fine, or as a tool in drawing software. The issue would have been, for the AI bosses, that this takes time and doesn't create trillion dollar companies. It only creates companies that they'd actually have to run, long term, and that's boring.
> Had LLMs first been presented as a "better IntelliSense", where rather than auto-completing function names, your editor can now suggest implementations
Aside from the AI dungeon app, this was my first real exposure to LLMs wrt github's copilot extension. It sucked, and did a lot of stuff wrong, but they got exactly 5 bucks from me to try it.
It's sorta crazy to think how many years ago that was now.
> The issue would have been, for the AI bosses, that this takes time and doesn't create trillion dollar companies. It only creates companies that they'd actually have to run, long term, and that's boring.
I feel like you’re not acknowledging the inevitability of the technology being developed further. You’re also ignoring competition and the basic fact that people do research and push the boundaries of their field naturally.
If the major frontier AI labs had decided, as you said, to make a fancy autocomplete and a better Grammarly and then simply stop and coast, do you really believe that every AI researcher and entrepreneur in the entire world would look at the situation and decide that the status quo is where it should end?
Humans don’t work that way. People would get ideas and want to push the limit. If the frontier labs stopped developing their technology, other companies would be created to carry on. Academic researchers would push the limits further. It would be discovered that the AI could go further and we’d get back to this same eventuality with a different set of people leading it.
I also disagree with your characterization of this as a get rich quick scheme for AI leaders who want to make a lot of money and then be done. I think this characterization feels truthy if your only conception of startups is as a gamble to retire early, but the people who build and run these companies actually do like building and running companies. If the AI leaders and executive management you’re talking about were separated from their jobs today, even with billions of equity, most of them would be out starting new companies next month because that’s what people like that do. I’m not saying this to “glaze” them or whatever I’ll surely be accused of in the comments, I’m stating some facts about this space. You don’t need to look far to find examples of this already happening.
I really don't like the excuses that are inherent to the "well it was inevitable" line of thinking. Oppenheimer was still responsible for the bomb despite it being "inevitable" in a similar way.
Despite this, when discussing the egregious behavior of the frontier labs, there is vanishingly little acceptance of the idea that 1) yes, it probably was inevitable in some ways AND 2) those involved, particularly at the top, are still responsible for the consequences of their (bad) decisions.
I am not sure what gripe you have with the labs, but I think if you are worried about AI then you got shockingly lucky with the lab leadership. My biggest gripe has been the campaign of negativity against ai.
I have also not been able to lean towards a side in inevitable v/s responsible.
1. Is it both inevitable and still they are responsible? 2. Or it was inevitable and they were just an instrument to make it happen. 3. Or it wasn't inevitable and if it were not for some specific people then it would not have happened?
I am leaning more towards 2. But if that is the case why are only specific people reaping the benefits of this?
If it is 3 or 1, then why aren't we holding them accountable?
"Inevitable" means "same technological progress would be achieved anyway, even if these people have made less damaging choices".
So yes, they are 100% responsible for damage they caused and did not had to happen. And no, "I had no choice, because it was the only way to become billionaire" is not an excuse.
They were NOT "instrument". No one manipulated them. No one forced them. Nothing forced them. They themselves wanted to and did not cared
“not acknowledging the inevitability of the technology being developed further”
This looks a lot like the “if I didn’t do $badthing someone else would have” line of reasoning.
This was the logic for developing the atomic bomb and it was a good call.
Was it? We nearly destroyed ourselves with nuclear weapons several times in the last century. I don't think it was a good call just because we've gotten lucky so far.
Academic research has always progressed and pushed the boundaries. Is that a $badthing to you?
Imagine a bag filled with marbles. We can't see inside the bag, we can just pull marbles out of the bag. Some marbles are white - they improve the world in some way. Wifi. Trains. Electricity. Some marbles are grey - they are useful but dangerous. Firearms. Nuclear weapons. Gene editing. And some marbles are black. If we pull a black marble out of the bag, we hurt or destroy ourselves.
Humanity's strategy so far has been to pull marbles out of the bag as fast as we can. And that's been working out, because most marbles are white. But we're insanely lucky we haven't had a nuclear war. The USSR's early warning systems picked up 5 incoming american missiles in 1983. They didn't fire back because one of the officers on shift didn't believe the reports (why only 5 missiles?), and he declared the reports were fake.[1] If a different person was on shift, there's a good chance nuclear war would have broken out because of a sensor glitch. And that's far from the only instance. The list is terrifying: [2]
And now we're improving AI capabilities as fast as we can, despite it being pretty clearly another grey ball. AI has a lot of amazing uses. But why are we rushing so much? Doesn't seem wise.
[1] https://en.wikipedia.org/wiki/1983_Soviet_nuclear_false_alar...
[2] https://en.wikipedia.org/wiki/Nuclear_close_calls
It's not about merely knowing how to build a "badthing". It's about knowing why not to.
With AI, the point is who uses it for what.
Think about it like a magic wand that makes your wishes come true. Would you really want everybody to have one? Who should and who shouldn't? Who gets to decide?
And this sort of blanket argument is how you get bad science on the order of Prison experiments to Nazi scientists. Yes, academic research that pushes boundaries can be a $badthing
You operate within the "logic" of capitalism, where no other value exists but the (short term) economic one.
AI indeed goes far beyond "autocomplete" and other nonsense. But the question is, whether it's a good idea to just let the "market" run wild with it.
Short-term capital interests don't align with the public good. AI greatly scales capability, but without anything to restrain it, that implies those non-aligned interests overwhelming society's defenses, outright destroying it in the process.
The idea of rich people being some kind of benevolent caretakers, who independently and dependably foresee and manage the consequences of their actions in everybody's best interest, is laughably wrong already.
With AI, that reality gap will swallow whatever is left.
> You operate within the "logic" of capitalism, where no other value exists but the (short term) economic one.
I operate within the logic of how the world actually works.
Which I find infinitely more useful for practical topics like the reality of AI development in a global economy.
Fantasizing about alternative realities where all the leaders of the world unite and halt technology development without any of them secretly defecting and developing it further isn’t very useful to discuss. These things aren’t going to happen.
The market is irrelevant. In the 28 centuries of recorded history, the human race has pursued power in it's various forms (political, military or economic), without regard to the "public good" (which is really, just a polite euphemism for the established order and those that benefit for it).
Without capitalism, you wouldn't be able to restrain AI adoption any more than the armored knights of the middle ages were able to restrain the adoption of the firearm.
Capitalism isn't the most palatable economic system, it's the one that best matches the reality of nature (competition)
I think the vast majority of the rich are greedy bastards, but IMO being rich does not imply greedy bastard 100% of the time.
> it's the one that best matches the reality of nature (competition)
This doesn't seem right: it doesn't match the history of reality, in which capitalism has repeatedly stymied rather than fostered competition.
> Had LLMs first been presented as a "better IntelliSense", where rather than auto-completing function names, your editor can now suggest implementations, very few people would have been upset.
I'm not rooting for AI companies or hustlers that are out to ride the wave and get rich.
However your claim is just hyperbole. I've used it in cross-cutting projects to extend protocol on desktop, iot device, which involves holding premise long enough to co-develop and test. It is also fantastic in debugging multi-threaded systems.
This is an interesting theme that I see a lot: Some people are firmly anchored to an idea of AI capabilities that is a year or more behind the frontier.
Some times it comes from people who try out ChatGPT free tier or the Google AI summary and think it represents the state of the art.
More recently I suspect it’s from people who build their opinion based on being contrarian and cynical: They dislike the AI people and distrust everything they say. If the AI people say it’s changing the world, then the cynical and contrarian belief is to assume that the technology is actually very bad and they’re just lying to make a lot of money and retire.
These takes were more understandable in the earlier phases of AI when it really was very limited, but it’s getting harder to understand how someone can be thinking of AI as just a slightly better autocomplete or slightly better Grammarly at the end of 2026.
> but it’s getting harder to understand how someone can be thinking of AI as just a slightly better autocomplete or slightly better Grammarly at the end of 2026.
That wasn't the point. My point was that if you had rolled LLMs out like that, as embedded technology in existing products. Slowly, over time, you'd have a lot less cynical people. I think the hostility towards AI product in large parts comes from the disruption and glancing over the negative parts. What we're doing now is the equivalent of putting millions of cars on roads which five years ago had never seem a car, only horses. We're doing so without any understanding of how that might affect society. Social media was a similar story. It was slower, yet we still didn't catch the majority of the negative consequences.
If it turns out that the people behind social media, and smart phones, don't care about e.g. peoples mental well being, then why would I trust that the people behind AI wouldn't behave the same way?
As I see it there are currently 3 broad classes of LLM usage:
1. enhanced autocomplete -- either line (e.g. JetBrains small language models) or longer predictions;
2. chat -- asking questions related to the task you are doing;
3. tool assisted/agentic -- letting models interact with tools to perform tasks.
I personally find 1 & 2 helpful for my development tasks. For 3, I'm cautious of using it for several reasons:
1. lack of control of what gets sent to the model (e.g. if you have confidential user data, proprietary code and/or data, access tokens/private keys);
2. letting the models run arbitrary commands (modifying the system, deleting files, etc.);
3. not having control over the design process, formatting of the files, etc.
There are people whose livelihood depends on them not getting it. There are people whose worldviews depend on them not getting it. Denial is a part of grieving. There are also people who like to troll the ai maniacs because ai maniacs are easy to troll. It has been curious to me to see all this as well, smart people who seem oblivious feels strange.
> Had LLMs first been presented as a "better IntelliSense"... very few people would have been upset.
Presenting AI as something it isn't doesn't change the reality which is like the Wait But Why thing https://waitbutwhy.com/2015/01/artificial-intelligence-revol...
That was written in 2015 before LLMs were much of a thing and has been fairly spot on so far. This is not just about LLMs any more than the industrial revolution was just about steam engines.
There are a huge number of people involved in AI outside of the major AI companies both in smaller AI companies and lots of independent developers. They aren't having much success bringing AI to the masses and this is because they don't care enough about money and they aren't selling their ideas aggressively enough. There is this odd tendency to undersell achievements because they weren't motivated by the Right Stuff.
The AI companies are as much being pushed along by investors as they are in the driving seat, but the push is in the direction of getting real products in front of people and experimenting with this miraculous tech everywhere it could possible work.
What would be the Right Stuff?
Ignoring the implementation details, if we treat the whole global technology world as one single AI system made up of AI companies, independent developers/researchers, and end users, what would be the training goal and evaluation?
> There are a huge number of people involved in AI outside of the major AI companies
How many, approximately?
better intellisense is how it started for me at least.
An excellent comment. It's very true - there's a deep sickness here which causes people to flee from the inherent value (relatively modest) and the longer, more tedious work of building a sustainable company around it.
> there's a deep sickness here which causes people to flee from the inherent value (relatively modest)
Value is simply the mutual understanding that something should cost a certain amount, and the system we've built emphasizes the need for profit above all else. When a set of companies comes along with a new, impressive technology claiming that they can automate away the white collar workforce, they're trying to capture as much value as they can.
It's not a moral sickness, it's market actors behaving rationally. Believing that people should be willing to leave money on the table because it's the right thing to do is a dead end, the entire system needs a rethink.
> It's not a moral sickness, it's market actors behaving rationally. Believing that people should be willing to leave money on the table because it's the right thing to do is a dead end
I don't agree that "I am just a market actor behaving rationally" somehow absolves you of any moral duty
I'm not an ethicist or anything but I have studied quite a lot of ethical frameworks and I've never encountered one that I think would agree with this
> Believing that people should be willing to leave money on the table because it's the right thing to do is a dead end, the entire system needs a rethink.
I didn’t expect a full throated defense of mugging and burglary on HN this morning, but here we are.
I enthusiastically agree. I've been on this ride since GPT 2 and 3, I did start out looking at the technology in a neutral or positive light. I did not think the stewards of this technology would be so irresponsible, its consequences would rival the Stalinist purges.
No matter what practical benefits this technology might bring, if this runaway marketing campaign isn't stopped, in just a couple decades we'll see what the complete collapse of civil society looks like, a world where the institutions no longer see value in maintaining an educated democratic populous, and can relax back into the comfort of feudalism.
I’m sorry, but the AI workers are as much to blame.
Or have you forgotten that Sam Altman was ousted by individuals that shared your concerns, and it was the OpenAI employees that rebelled against the board and wanted Altman back?
That HN post was infamous and nearly took down this site.
The AI formula is out of the bag. The Manhattan Project is over, trinity has happened.
If it weren’t OAI and Anthropic, it would be others.
If it weren’t US companies, it would be others.
It is far too useful now to stop large scale deployment, it isn’t a matter of specific people or countries not being cautious or prudent enough. The entirety of the Earth is at issue now and there will always be people who cause the tragedy of the commons (if indeed AI turns out to be harmful in the long run, which is not remotely certain).
Look at what happened with fossil fuels. Even if the entire global west totally ceased use, there are many countries that simply will not stop because of local utility. AI is the exact same way.
> If it weren’t US companies, it would be others.
There is something deeply rotten about the US mindset though, them being American is definitely a problem.
> Look at what happened with fossil fuels.
The west had a 100 years head start, on top of that these countries are producing everything you consume, don't start pointing fingers. Chinese current co2 per capita is the same as the US in 1900
> There is something deeply rotten about the US mindset
What is that “something”?
I remember reading about the development of neural networks in a magazine from the 1980s.
Releasing artificial intelligence for the use of all of humanity is better than it being controlled by one or more shadow groups. That's what the Americans did, and the Chinese followed in.
Had AI been invented in the 20th century, it would have been kept secret and used only by state actors to facilitate genocides at mass scales. But we live in different times.
How can "AI" facilitate genocide? Like what might chatgpt have suggested that would have made the nazis more effective or the khmer rouge.
Widespread mass realtime facial recognition usage is allowing governments to target and repress subsets of the population with an unprecedented scale and efficiency. ICE are flying facial recognition drones over sidewalks in downtown LA. Dissdents are having their facial geometries entered into secret databases simply based on political opinions.
Any new technology throughout history has first been used for war and genocide.
How could modern AI have helped those groups you mention in their endeavour? Really fucking easy and effectively, and you don't need a lot of imagination to think of how. From surveillance and interrogation to pinpoint determination of ethnicity and much more. Not to mention translation between human languages.
> Releasing artificial intelligence for the use of all of humanity is better than it being controlled by one or more shadow groups. That's what the Americans did
Uh? In what universe are openai and anthropic not shadow groups?
Are you misunderstanding on purpose? OpenAI and Anthropic let the public use their AI models. Shadow groups would mean that people wouldn't even know about the existence of AI.
> OpenAI and Anthropic let the public use their AI models.
Oh, how nice of them, such charitable souls...
Cool snark. I said it's better that all humanity have access to use AI rather than just a few people. That's pretty nice, compared to the alternative.
But if you're an angry hacker in full on culture war witch hunter mode, then of course nobody can say something that's even slightly different than full-on hate without you interpreting that as aligning with the evil witch.
Yes, the cat is out of the bag as to how to create this type of AI.
But that doesn't mean that everyone has to use it.
We're only starting to see the beginning of the negative impact of AI to the job market and society. If if turns out that allowing most jobs to be replaced by AI is bad for society and the economy (super high unemployment - nobody buying more then necessities), then wouldn't it be smarter to not do that?
If that's the case, then each country can decide for themselves.
Dario's "Machines of loving grace" UBI/foodstamp "utopia" sure doesn't sound like a recipe for success.
Which job are worth doing and which ones you'd rather the AI do? AI should enable things that were previously impossible because of time constraints. If time is money and we were previously trading with money, we will get back to trading time. Manufacturing time for instance. Compute times. For commodities, that might be automated to some extent, I am not sure. The friction in exchanges is where the money is made and the friction here may be at the interface with the real world still.
> If it weren’t OAI and Anthropic, it would be others.
Doing what? Rushing to make trillion dollar companies, rather then developing the tech and their companies in a sustainable way? I don't even know what in the original comment this would be a response to.
Rushing towards creating machines that can give you vast amounts of military, political or economic power without regard to it's effects on the old order.
The particular form of the pursuit of power practiced by OAI and Antrophic - the creation of trillion dollar companies, is one of the most benign in recorded history. It's so unnaturally benign, that I doubt it's going to survive for much longer.
This is getting boring. Either go all the way, RSI, singularity, whatever or pull the plug on the whole thing and shut down all data centres.
"he warned it could trigger a large-scale incident this year or next, such as a temporary blackout of financial or power systems."
I know this sounds unusual, but I do think this wouldn't be the worst thing that could happen. Sure, it would cause a bit of chaos, but it might also make us think twice about continuing down our current path of unfettered development, and possibly even make us much more cautious, which might in turn even help us with the climate crisis.
Because human beings in a large mass only respond to one thing: immediate difficulty. They don't respond to delayed difficulty like "the climate is getting worse" or "technology will make life more difficult in the distant future". They only respond to immediate pain.
So I think it would be a good thing if the financial system had a serious threat. Then people with money, who tend to offset future problems with technical solutions and financial padding, would start to become much more suspicious of technology and defund AI and wasteful things that are likely to threaten their safety.
I agree with your premise, but not your conclusion. The problem is not the scope or scale of things, but treating them in an absurdly stupid fashion. Why are power systems internet connected? It certainly enables some greater degree of power and flexibility, but it exposes them to potentially catastrophic outcomes in case of malfeasance. So what happened at some point is somebody convinced somebody else that the security was super duper ultra good, and that's simply not true.
And I'd go one step further to say that it probably can not be true. This has always been true and probably will always be true. All LLMs are doing is demonstrating that. And the LLMs of tomorrow will demonstrate that the most LLM hardened system of today is still just a vulnerable mess. So the conclusion is simple - things should not be connected to the internet unless the worst case scenario from hacking/exploiting is relatively low. If somebody wants to connect their coffee maker to the internet, for one can only imagine whatever reason, then the worst case scenario is a broken coffee pot - who cares, go for it. But connecting e.g. power, water, or other core infrastructure to the internet? No, never.
Just because something hasn't been proven to be a problem today does not mean it won't be a problem tomorrow. We shouldn't learned that yesterday, hopefully we can learn it today.
> Why are power systems internet connected?
Are they?
We are starting to get a taste. Credit Mutuel Arkea just admitted having been hacked. Over 100 millions gone.
Blockchain despite all its flaws, is the canary in the coal mine. They have been dealing with an increasing amount of cyberattacks. Because it all happens publicly in the blockchain, there is no way to hide it.
However major bank gets hacked in the tune of over 100 millions and we get to know it happened because somebody who has no relationship to the hack turned his eyes to a bunch of suspicious transactions happening on Ethereum.
Blockchain dapps were getting hacked long before AI. Not a signal
If people's pensions & life savings are flushed out the back door (literally), rest assured attention WILL be paid.
Don't ignore the very real human cost this would bring. Every day the power is out, more vulnerable people would be harmed(including death) directly from not having electricity for essentials or from systemic failures like in food distribution and storage. Financial systems being down could also have extreme impacts on some people, but maybe easier to mitigate via human compassion.
Maybe the long term changes such blackouts would cause would be "worth it" in a utilitarian sense, but let's not hope for such damage.
You’re describing a weapon, and if AI managed to do that (I believe we are not far away), then this would probably increase the investment not decrease it.
Would it? I don't think so. Not if the weapon is an indiscriminate one and makes everyone worse off. Yes, some people (maybe military generals) might support it. But there would be near universal hatred for it, if people wake up one day and find their bank accounts not working or their eletricity not working.
This is why I think AI is so stupid. It's just too powerful for anyone to control, especially with the inertial mass of capitalistic society powering it through the incremental, short-term advantages it gives.
> Not if the weapon is an indiscriminate one and makes everyone worse off.
Nuclear weapons are indiscriminate and make everyone worse off if used. And everybody has them. Your optimism is unfounded.
> Nuclear weapons are indiscriminate and make everyone worse off if used.
The difference is most people have not experienced any immediate effects of nuclear weapons. But people will experience the immediate effects of AI taking down the electricity grid, for instance.
Right, just like the war to end all wars totally prevented the nations involved from going to war with each other... for a grand total of 20 years before they decided to do it again. There are other more powerful and influential factors that decide the fate of nations. Even if you get your wish and everyone suffers and everyone hates AI then it will simply disappear into classified research projects and government run data centres. The NSA could easily absorb it but I'm guessing government contractors would also want a piece of the pie for weapons research.
If we're at the point where all that matters is scale and reinforcement training then the public facing aspects could entirely disappear and progress would still continue especially when given an infinite budget of taxpayer dollars.
"most people" are in the set of things the trillionaires don't give a shit about. Their market distortion has already made PCs unaffordable and electricity prices rise, among other adverse effects. What else has to happen so you take notice?
To add to your point, many people’s jobs had been affected by AI, job market data don’t lie. That didn’t stop the progress, it became exponentially faster.
I agree that people only really respond when something goes wrong badly enough that its impossible to ignore.
I disagree that something like this would be a good thing though. Whether it's a power outage, financial crisis, or something else people will die. If that's what happens and society ends up adjusting course after then that's what happens, but I can't call people dying a good thing.
It would be a lesser evil rather than good.
This sort of thing was going to happen, with or without AI (state sponsored hackers, the US or China taking down systems they control, or whatever).
The later it happens, the more we will be down the path that lets it happen, and the worse the consequences. We are getting gradually worse consequences: AWS outages taking down websites, GCP outages taking down banks, attacks on fridges disrupting good supplies...
Its better for us to change path when something happens that causes dozens of people die rather than wait until its millions.
> people will die.
I don't think that's a good thing. But if we DON'T do anything, then even more poeple are likely to suffer. So while people dying is not a good thing, a thousand people dying is better than a million people dying because we don't do anything.
Same thing about burning your hand. It's not a good thing to burn your hand, but burning your hand once when you're young probably saves you from a more serious accident later on.
We were probably better off before agriculture, but here we are.
So Hinton doesn't trust the tech leaders but then the consequence of having governments censor LLMs is that only tech leaders and governments have unlimited access to AI.
If someone manages with the help of AI to hack into a vital system, then the system was already in a critically bad state beforehand. If developers can't use AI to check the code for security flaws, then security gets worse and not better.
If you are stuck on censorship you are just way outside the window of what is actually of concern here...
I would posit that systems being hacked by AI were already vulnerable to the huge caches of secret zero days held by intelligence agencies and organized crime around the world. Which we know exist.
Insecure is insecure. These systems never were secure.
Of course, we are. Back in the late 80’s early 90’s, no one working on ML was aspiring to develop systems which would be used to generate the kind of AI crap which plagues us today.The good is being smothered by the absolutely soul-crushing pointless.
AI exposes the Sturgeon's law (90% of everything is crap), and amplifies it much more, and anything good is drowned out. Most of human-generated popular music is slop. Most of human generated code is buggy, unmaintainable and full of debt. Social media and youtube have always been dominated by click-bait and mediocre time-wasting crap.
I hear you and i find it devastating, too. On top of that, the negative effects on the planet for having this technology vomit this soul-crushingly pointless crap is just depressing.
Before AI I was already being smothered by soul crushing pointless junk made by humans.
You’re placing blame on the wrong spot. It’s algorithms and social media incentive systems not AI. Our digital media incentives addictive minimum effort chum and slop. AI just makes it easier to make.
> temporary blackout of financial or power systems
I'm reading this in the context of the article currently at the top:
AI is forcing us to pay off 40 years of technical debt
https://news.ycombinator.com/item?id=50364336
Is it bad we're finding all the bugs now? Or is it good that we're finally being forced to make our critical systems actually robust?
"We". Who "is" we? This isn't about you. or me. there was never a we. Before you open your mouth or put your pen down, wouldn't you think of the shareholder value for once?
My main fear is that the integrity of everything will be compromised, but everyone will keep acting as if it's not. I feel like it's already somewhat the case but I'm worried about it getting worse.
My prediction is that some people will be able to use AI to create essentially infinite money inside their bank accounts (through hacking); the prices of assets will grow to ridiculous sums as the people with infinite money buy up assets, but yields on all assets would soon approach 0%; still, one asset is going to grow at a faster rate than all other assets (E.g. an AI company backed by blockchain/tokenized shares that are impossible to hack) and everyone is going to start piling their infinite money into that asset; every other asset will crash because of the rapid, exponential price acceleration of the top asset making every other asset unattractive by comparison. Then fiat money would cease to exist and shares (or crypto tokens) of the top company will be used as money instead. Then prices of assets would stabilize again; denominated in the new currency... Then after that, we will finally be able to start innovating with AI.
I suppose, but precious metals and cryptocurriencies would still exist, and would probably be more immune to power struggles and hacking than stocks.
Not to say that they're unhackable or unstealable, but merely that they dont have a central weak point that if broken, destroys the whole system
I also think so. The weakness of current popular cryptocurrencies though, is the complexity and size of their codebases. Many above 100k lines of code. That's risky. Though the benefit is that they are somewhat heterogeneous (different nodes running slightly different code/versions) so it may be hard to find a vulnerability which affects more than 50% of nodes; and this is a hard requirement for compromising the integrity of that system.
Traditional finance has the benefit that they can more easily reverse malicious transactions, but if hacks become too common, it will be impossible to correct fast enough and in a reliable way. With traditional banking, a hacker only needs to hack a few nodes to compromise the integrity of the entire system.
So yeah I also think crypto will win in the long run. I would bet on a tokenized stock with AI attached.
I think the top asset will combine the AI which is best at hacking with the blockchain which is hardest to hack. Ability to corrupt others whilst maintaining perfect integrity yourself.
AI for offence, blockchain for defence.
That's weird, they must have cut off the part where he said Sundar.
> What's risk for people valued close to or over trillion dollars?
Mortality.
Does it increase the more business risk these folks take?
Speak for yourself - I'm in AI heaven.
Programming has never been this awesome.
Yeah, seriously. AI has been the single biggest technological improvement to my life in the last 20 years. If I could choose between smartphones and AI, I'll take AI any time of the day. Smartphones (in combination with social media) had a worse impact on society than AI has had and are not worth their advantages to me.
The people in wall-e were in AI heaven.
Can't agree more - some of us are just happily achieving our life goals in record time thanks to AI!
There's just something off about this mindset - why is achieving your life goals faster better? I think you haven't realized that life isn't a bunch of checklists you need to complete but rather about the journey, and when you do get that, you don't need to go fast.
Also the future where there is no sacrifice to achieve your lifegoals seems just unnatural to me.
Such a smart guy so late for the party. Thanks to him and others who understand a lot about tech, we gave monster a weapon to destroy us.
I still don't know how Karpathy still works with those folks. I really think the guy is great but he makes me wonder
Calling for Off-gird mesh communication
The same applies to most of humanity’s other inventions, such as transportation, weapon, electricity... Ultimately, these technologies serve as instruments for rivalry, deciding the fate of the strong versus the weak.
As is the case with any technology in the hands of careless people, but unfortunately we can't wait for society to catch up morally to the technical standards, innovation is organic and breakthroughs have no timing.
> Hinton, 78, said that within a decade and possibly sooner, humanity will know whether it can manage the technology’s risks and enable what he called “radical abundance” or whether “we fail to manage the risks and … have a huge gap between rich and poor, huge cultural problems, massive unemployment, just society being torn apart”.
Uh... we're already at the latter prediction.
"It might turn out we were better off not opening Pandora's box" says man who helped open Pandora's box.
That's a touching sentiment or whatever but the nature of the thing is that it will inevitably be opened and that it cannot be closed thereafter. So (I think) we'd all be better off putting our efforts towards predicting the likely outcomes and figuring out how to make the best of them.
AI winter has arrived.
Said AI "experts" for the last 10 years.
Nobody has said that. The winter will probably come, but it's not at all clear when. First the bubble must burst.
Nobody said that? The GP literally said it!
Why are the using the Ukrainian transliteration of Chernobyl now
almost every story that has ever been written about a world with AI has posited that humanity was better off without AI.
I think we did good inventing it and it's inevitable. Lets use it well.
modern day Mary Shelley
We are not but we need a better approach, https://www.ethicalacceleration.org
> It could help cure diseases, end boring work and give us more free time.
Seriously? It seems to be causing more problems, giving us more boring work, and less free time.... sounds like absolute bullshit.
"Could" does a lot of work in these kinds of claims.
We need to build it as fast as possible because it could cure all disease. Ignore whether that's possible, how it might happen, whether all we're missing to do it is more intelligence, etc.
The only way I see us curing all disease is to throw out all the names we've given to collections of symptoms. What disease? I don't see no disease, you just have a sore throat, runny nose, and mild fever.
> We need to build it as fast as possible because it could cure all disease
Hell no. We need to risk pumping out even more CO2, causing more environmental disaster, ruining entire human communities, just to cure diseases? You might as well say it's better to be in prison because you're guaranteed three meals a day. The logic of some pro-AI people is baffling.
How has AI given you less free time?
I have to spend extra time just trying to find AI-free things, AI-free writing, AI-free communities, and people who aren't addicted to AI. And get noticed/get a job by having to push through all this AI static. And having to interact with AI-based customer service agents that take 10x as long to resolve the problem as a human being. And having to wait a bit longer because there are 10x more 'Are You Human' Captchas slowing down website loading. And having to download OS updates that are 2x as large because they contain AI models. And having to double-check everything because there's so much more hallucinated crap out there.
The other day I hired an accountant and even he sent me an AI answer that sounded right but was actually bullshit.
In short, it takes a LOT more time just to find some sanity now.
AI is shit.
I mean, it could if "we" "just" got out of globalized financial capitalism, changed everything about our political, economical, financial, social systems, etc.
Only 14 years old and accelerationists think we can just push the capitalism pedal further and somehow get social changes going their way
I don't think accelerationists think we can do anything. The thought process of the accelerationist is partly fueled by the 'excitement' of betting and playing musical chairs and hoping to be the first ones out holding a bag of money and partly fueld by the delusion that technological advancement is always good at any level of complexity.
That's the whole point of the site. Did you read it. AI should not cause harm, direct or indirect.
Of course I read it. But my thesis is that one cannot avoid that. Causing harm, whether direct or indirect, is the fundamental nature of AI.
> Causing harm, whether direct or indirect, is the fundamental nature of AI.
How so?
It can be avoided, humans control AI still. Humans are the ones causing harm with the help of humans.
You’re a cultist
How did you come to that conclusion?
To be honest, I think it depends on the environment. In the past, if you didn't come out of a good university, your access to knowledge was limited. But these days, in terms of access to knowledge, I think things have clearly improved, whether or not AI is illegally sharing knowledge.
Looking back now, was it really happier before AI existed? I'm not sure. In the past, when I wanted to find out whether I was right or wrong about some piece of knowledge or information, I'd contact several people and, far from getting an answer, I'd often be told not to bother them. But AI, in that respect, is the best teacher I can think of.
I think the times are continuously getting better. It's just that there's a problem of power concentration among the people who handle it. Still, if AI is a tool of that nature, then I suppose that can happen.
But considering what access to knowledge was like before AI came along, I think having AI is better.
I'm not sure of this. I remember back in the day finding people's blogs, reading old forum threads, participating in said forums - there was a ton of info out there and easily found. Now those kinds of sites are rare, and those that still exist aren't surfaced by search engines like they used to be.
AI has the benefit of having trained on all that material, it's a snapshot in a sense. But as people increasingly stop creating content like that and we turn to AI instead of asking for help on forums we stop growing the knowledgebase that AI draws from.
Knowledge was already extremely accessible in the age of the Internet, but it required time to read and learn. AI is making that same knowledge applicable without the prerequisite of learning.
Most people are not interesting in learning for learning's sake, they are motivated to learn something they can apply (see: kids complaining about how learning Math in school doesn't transfer to useful skills). Why learn about architecture and technical debt, if you can tell the machine to make it faster?
I don't think you're wrong, but fundamentally we're in an age of information overload, and this may be the new normal. In other words, the environment where you had to know things deeply and become an expert, like in the past, may be different from an environment where you know many things broadly and combine them.
I basically don't think you're wrong. However, books and reading have their information fixed at the point of publication, whereas depending on where you live or your community, the information you get buried in differs. Because I was in an Asian country, it was actually very hard for me to synchronize my programming skills with the latest technology in the US. The frameworks commonly used in my country are stuck at roughly 15 years ago. The time lag between US technology and my mainstream technology is about 5 years. In my area, it's about 15 years.
Anyway, thinking about your opinion, I think there are both valid points and dark points. I think learning for its own sake is important. I agree that people should have an attitude of learning for its own sake, but I don't think it's mandatory. I think it should vary depending on the environment. Still... I do think having that attitude is a good thing. I like your idealism, senior.
However, that holds for people who are middle class or above and can survive. Because rich kids can picture the future and are given the right to love learning. But if you're poor, that's very hard. And when you grow up in such a poor family, there's an emphasis on it having to lead to practical skills.
And even if someone loves learning for its own sake, I think quite a lot of people love the honor that comes from learning. But that too can be called loving learning itself. It's purely a matter of individual values.
I don't think you're wrong, senior, but I also wonder whether the old ways are right in today's era of excessive information. It's just that because our lives and the environments surrounding us are different, we came to different conclusions. I upvoted because I agree with your thoughts. But there are points where I differ. Have a good day.
The internet pre-AI did far more to make knowledge accessible than AI has done. Its useful, but its not the critical thing.
What is the point of this quote? Or any of his quotes for that matter.
I guess capsule network haven't improved ...
I actually laughed out loud.
I do find capsules interesting though.
We should introduce regulation so only credentialed software engineers are allowed to use AI.
I would rather see the onus on the company owners to be responsible for what they sell. Serious penalties for seriously bad products.
Absolutely! And software engineer should be an hereditary title as well.
In East Asian countries, certification systems are still maintained, but in reality they're practically meaningless. If a certification system exists, people will take an interest in CS and work harder at it, or so you'd think—but in reality, the exam questions end up stuck in the old style. In other words, there's a big gap between the exams and the actual field.
Because a certification has to be fixed at a particular point in time. Korea's certification exams start with the 23 GoF patterns and go on to rote memorization of countless patterns and layered terminology.
The problem is that the way you learn for this certification is the old way, so someone who gets the certification ends up writing legacy code.
But if you always update the questions to the latest, then nobody can pass the certification. So, having seen how the certification system works in East Asia, I don't think it's all that good
We should have not left the trees. You cannot disrupt banking and power if they do not exist. Checkmate!
The ghost of Douglas Adams has entered the chat
It might turn out we were better off without nuclear weapons, but so far I’m happy we haven’t had another world war (in the traditional “total war” sense).
No way to tell until after, though. So far, mutually assured destruction is working. It works until it doesn’t.
Same with AI. Thus far we’re better off with it than without it. That may change.
I mean, look what we did with fossil fuels. Some people are even still willingly burning them, even though the data is clear that our use of them has already started the total destruction of our civilization.
Is there a name for this, a technology that is so insanely useful that there is no way we won’t exploit it, but where the worst case scenario is total collapse, and impossible to determine well in advance?
(Fossil fuels I suppose don’t count as even the energy companies knew unambiguously decades ago that large scale use would eventually bake the entire agricultural system out of existence.)
>so far I’m happy we haven’t had another world war
The reason for that was more that the 3-4 great empires involved in the previous two were exhausted, Japan was defeated, China was in no shape, and US vs USSR had no reason to push for one, since the Cold War era gave them plenty of breathing room, roughly half the world each.
> roughly half the world each.
Both the USSR and US fought multiple wars directly against the proxies of the other. They weren't simply hanging back and letting others do the dirty work. In most any other era one of those would have become a full blown war between the two. Both sides pushed the line of not directly involved and then both sides looked the other way versus escalating. Nukes made the alternative too much of a risk for both sides.
oh word?
The scene is bleak, silent, desert-like with a few dead trees bleached white by unfiltered UV. Nothing moves. In the distance a cave, and from that cave
... no shit Sherlock
9 minutes. Was that enough time to read the article?
My reader mode says 6–7 minutes, so, yes.
TFA could have been two sentences:
He warns that AI might cause serious problems: power failures, job losses, and greater social inequality.
He believes AI might improve our lives, but governments must make sure it is safe.
Did you look at it? It wasn’t that long.
I personally feel reading the article before commenting affects the purity of my vibe before posting
vibe comments!
So why did he start to develop it in the 1970s?
Why did he start a research direction that 50 years later "might turn out" to not be a good idea?
That's a relatively silly question, don't you think?
No it's a completely serious question. The goal has always been to replicate human/biological thinking processes in software, and now that this has been more or less achieved he's suddenly not happy about it. There's been plenty of opportunity to actually think about "what if all of this actually works" but they chose not to.
Sounds like sour grapes. He worked on it for a long time but he doesn't own a trillion dollar company. Missed the boat?
This is a bit like Rudolf Diesel saying maybe we're better off without airplanes. The field has long since left this guy behind.
So pumping CO2 into the atmosphere is a good thing and if someone does a backflip by recognising their mistakes that is a bad thing?
All economic activity emits CO2 directly or indirectly. The question is are the benefits worth the cost, and could the process be more efficient. A molecule of data center CO2 doesn't affect the climate any more than one that comes from producing cement, or getting a Labubu delivered overnight. If we want less CO2 emitted, it is best to treat emissions equally and let people decide what they value when they spend their money.
We should regulate data centers to require fully renewable power (solar/wind/hydro + battery).
This is tangential encouraged by your point and other forays here. But one does wonder what arguments politicians (the power lies with them) understand when they make their pronouncements. The UK has for instance, prior to the new administration, totally and vociferously supported net zero. From the point of view (a fact for many but not for all informed authorities) that climate change is caused by CO2 and we are headed for a disaster - Net Zero is a logical step to trying to save humanity and the whole ecosystem. Now (November) I read that the UK powers that be, appear to have put it aside in actions at least. Countdown to Net Zero doesn't matter any more; open the oil fields, get rid of subsidies for photovoltaic and wind - and other measures. My point is not a political jibe but a stance I suspect to apply in other areas. Is climate change going to be a catastrophe or not? They should make up their minds.
If the resulting issues are correct it doesn't matter whether the field left the guy behind or not.
He would still be right and the field practioners would be wrong.