Linode Security Incident

(status.linode.com)

176 points | by lacoolj 8 hours ago ago

97 comments

  • 6thbit 7 hours ago

    It feels like we hit an inflection point in the last week or so. Kind of a large tidal wave to crash on the coast, but likely not the tsunami yet. I hope.

    If you are running services and don't yet have an easy way to react to these things with swift process defined, I hope you don't like to sleep.

    • joecool1029 3 hours ago

      > It feels like we hit an inflection point in the last week or so.

      This isn’t a new thing for Linode. A customer service panel vulnerability in the past was the reason Bitcoinica got hacked in the early bitcoin days: https://spectrum.ieee.org/thousands-of-bitcoins-stolen-in-a-...

    • FuckButtons 6 hours ago

      When mythos was released - I guesstimated that we had 6 months until hackers had similar enough capabilities that we would start to see real problems in the wild. Unless there is some change in the trend line things are about to get really quite spectacular.

    • rpcope1 3 hours ago

      I'm just waiting for someone to drop a real zero day on openssh or openvpn for before the real pants pooping starts.

      • gtech1 3 hours ago

        If you run openssh open to the public...

    • keyle 2 hours ago

      Tidal waves are one thing, this is a tsunami.

    • lifeisloving 7 hours ago

      Its really not that hard to secure your servers, especially if your not a high value target. If you are a high value target, you should have security professionals on staff.

      I dont think my security posture is all that much different than it was last year. Use a Secret Manager and or encryption, have backups, strong firewall rules, use canaries of some kind.

      If anything llms are making security worse because there's more people offloading their thinking on llms, or people that skipped some steps in their learning journey because they just ask the llm to do everything. I dont think its necessarily because attackers have llms now, but that contributes.

      Edit:

      Damn Im getting mega downvoted for saying this. So are people of just at a consensus that there's nothing you can do? That we're in security armagedon???

      Have you ever thought to consider that security took hit along with the general degradation of software because of AI??

      • moscoe 7 hours ago

        You are literally the everything is fine meme. I admire your confidence. I never realized it was so simple!

      • UqWBcuFx6NV4r 5 hours ago

        Is this like when people say “it’s not that hard to write memory-safe C”, and then nobody does? I take this as actually meaning: “I feel that I can keep my shit locked down.”. Whether or not that’s true, which i’m skeptical of, plenty of OTHER people can’t, or won’t, or whatever. That’s the reality.

        To speak to your edit: this is false equivalency. You can still decrease the odds of being hit. There are still better and worse ways to address security.

      • mort96 5 hours ago

        Can you tell me what I should be doing differently then? I just host some services, among them Nextcloud that's written in PHP, protected by SSL, on a server running Linux. And I feel like the past week has been a lots

      • tbrownaw 4 hours ago

        It's tedious, which is a particular kind of being hard.

      • sanex 6 hours ago

        I agree but disagree about the cause. People have always been bad at security and code in general.

      • amelius 4 hours ago

        Have you not heard of supply chain attacks?

      • bvcp 5 hours ago

        if anything we should be using ai to find this stuff before the bad actors, too bad the model providers are hell bent on regulatory capture via fear

  • spydum 7 hours ago

    Wow, that isn't clear. Is there an incident, or did they just become aware about MPTCP vuln? Details on the mptcp are also pretty scarce as well right?

    • csnover 2 hours ago

      It is almost certainly the latter, since it is another “Customer Security Advisory”. There is a currently open report for a tranche of nginx vulnerabilities[0] and a closed report for PwnEscape[1] and the PHP RCE[2] that are identical. The HN topic has been editorialised.

      [0] https://status.linode.com/incidents/8dqqkh4d9y2c

      [1] https://status.linode.com/incidents/zp71823t318w

      [2] https://status.linode.com/incidents/wq8zlhk736fk

    • 6thbit 7 hours ago

      Yeah its unclear. Sounds like they are just now patching it and treating it as an incident which is probably the right thing to do.

      • onei 7 hours ago

        (Former akamai employee) Akamai have an incident process that handles both "things are bad for multiple customers" as well as "we need eyes on this from multiple departments/stakeholders asap". If you're hitting a wall debugging an issue on your own, especially if it's not overtly customer impacting, declaring an incident is an effective way of getting the attention needed to solve it.

        Once you declare the incident, the process includes these status page updates, alongside equivalent internal status updates.

        In this case, they probably wanted to maximise coordination, so used the process designed for that.

        • CoastalCoder 6 hours ago

          Also a former Akamai employee.

          I left because I didn't like the pressure / pager duty, but I seriously respect how smooth and well thought out their operations were.

    • dofm 7 hours ago

      I am wondering if there is something significant written with Go in their system stack (Go uses MPTCP by default on listening sockets, it seems)

    • lacoolj 7 hours ago

      Yeah agreed. I actually only noticed because my two boxes are super slow and I've been troubleshooting for about an hour. Finally opened up the Customer Support page and this was the top incident.

    • tbrownaw 5 hours ago

      I haven't seen any emails about it, so I'd lean towards it being something that doesn't rate sending emails.

    • dboreham 2 hours ago

      Based on email I got from another big hosting provider, they're all patching for the same checksum metadata vulnerability (the CVE they mentioned is the same). There is however a root RCE PoC according to the researcher so...potentially bad.

  • paaloeye 7 hours ago

    Even better CVE 10.0 has just dropped https://news.ycombinator.com/item?id=50310937

  • nout 7 hours ago

    All companies should at this point invest a lot of money into defensive AI. In ~July the AI hit crypto and bitcoin, where it's the easiest to extract the most money. Now the AI powered attacks are hitting infra and SaaS. Next (~December), it will hit classic infrastructure, banks, small businesses, etc. By mid next year, we will be over this hump with defensive AI being required part of everything.

    • jeremyjh 7 hours ago

      Its already happening to classic infrastructure, small and medium size businesses. Absolutely anyone can summon a gremlin capable of exploiting stuff like this or the PHP thing last week. There are clankers probing this stuff all over the place within a day of it going public.

    • throwuxiytayq 7 hours ago

      I find it breathlessly funny how people think it's going to be AIs vs AIs fighting it out with light sabers and plasma lasers in the cyber space or some shit.

      Your intuition that this can be balanced out is wrong. When you have a CVE in your TCP stack, your vault unlocks and you're fucked, it's done. My dude, there is never going to be AIs scanning every incoming packet for signs of a 0day, and deciding whether to, uhh, cut it up with a lightsaber and shoot it with a laser pistol. Your options are to have a stack so secure that it can't be targeted, or to disconnect all network cables.

      • ForHackernews 6 hours ago

        I assume that "defensive AI" means finding and patching vulnerabilities before attackers do, not some kind of live LLM packet-inspection.

        It is possible to write more secure software[0], companies just choose not to prioritize security.

        [0] https://cr.yp.to/qmail/qmailsec-20071101.pdf

      • dist-epoch 6 hours ago

        AI's can decide on the action to open the vault file. It can also prevent TCP connections from unexpected IPs.

  • ericd 4 hours ago

    Anyone have any thoughts on what happens when banks/brokerages start getting hit in earnest? Think it's prudent to split single brokerage portfolios across multiple? Or do they have enough speed bumps in their procedures that they're not quite so vulnerable to major losses?

    • pianopatrick 3 hours ago

      If the large and politically connected banks and brokerages have losses large enough to threaten their survival as institutions then the government will jump in to help secure them and bail them out if needed.

      I would stick with a "too big to fail" institution instead of splitting across multiple.

      If you want a real "backup" you could ask for physical stock certificates or get physical gold or something else that cannot be taken from you by computer hacking.

      • pixl97 3 hours ago

        How is the government going to help secure them? Do they even have enough qualified people?

        • pianopatrick 2 hours ago

          Well I assume there are some smart hacker / security people in the NSA etc. who could be parachuted in to help if it really came to that.

          I think these banks will probably be able to hire equivalent people in the private sector though and might not need the government help.

        • spacephysics 3 hours ago

          FDIC insures deposits up to 250k. Big institutions basically all have this government backing. If an incident occurs, FDIC has deposits available next business day in event of a bank failure

          • baobabKoodaa 2 hours ago

            FDIC insurance is mostly unrelated to this. If someone hacks a brokerage, sells a bunch of customer assets for cash, adds new bank accounts for those customers, and then withdraws that cash to those new bank accounts, FDIC insurance does not help at all.

            • ericd 43 minutes ago

              Another bad one I've heard, buying thinly traded penny stocks as a way to exfiltrate money from brokerage accounts without wiring.

  • elashri 7 hours ago

    @dang I would encourage you to really think about having security incident as a new HN tag. The wording can be better though.

    • rglover 32 minutes ago

      [whoops]

    • downrightmike 7 hours ago

      We can call it [SI]

      • entrope 7 hours ago

        As long as we know that doesn't mean Systeme Internationale, Sports Illustrated or Super Intelligence.

        • gk1 7 hours ago

          > Super Intelligence

          I think that was the joke.

  • Arbortheus 7 hours ago

    This reads to me that nothing has happened and they are just trying to patch stuff? Right?

  • dist-epoch 6 hours ago

    I've lived to see the day when half of the Hacker News front page is about hacks. What a time to be alive.

  • malloryerik 2 hours ago

    Are banks safe?

  • nodesocket 4 hours ago

    Anybody recommend a CVE alert self-hosted app? E.G. I specify my software, OS’s, and versions and when a matching CVE is created I get an alert. Don’t want enterprise security products.

  • phendrenad2 3 hours ago

    Why was the previous discussion of this flagged? https://news.ycombinator.com/item?id=50299182 Is our new policy to stick our heads in the sand and pretend that CVEs don't exist? Because I'm not getting behind that!

  • optimalsolver 7 hours ago

    I mean, this situation is untenable. HN's front page is just going to be an endless list of breaches.

    I don't see how web services proceed like business is normal.

    • peri-cl 7 hours ago

      Introducing: HNN (Hacker non-news): a list of web domains *not* currently under apocalyptic AI attack. Shorter and more parsimonious.

      - Tell HNN: pets.com was *not* owned today

      - Tell HNN: about:blank was safe, too

      - Show HNN: my new gopher protocol project—no currently known CVE 10's

      • mceachen 7 hours ago

        gopher goes to 11

        (archie and veronica are non-denumerable)

    • LtWorf 7 hours ago

      Better than the usual endless stream of what your agent did today.

  • ForHackernews 6 hours ago

    What's the mitigation for this? There's no patches available yet https://ubuntu.com/security/CVE-2026-98432

    • markasoftware 6 hours ago

      Mitigation is to retire and wait for the singularity.

      Don't use MPTCP is the first mitigation, but my understanding is this might be exploitable without MPTCP.

      • bombcar 6 hours ago

        Filtering packets with ip-options set may also help, but it seems Akami is already doing that at the edge.

    • netrem 6 hours ago

      Debian does have a patch up

  • rvz 7 hours ago

    So it begins. The war against the abliterated AI models. Even at 1,000 TPS any service is a target.

    Imagine swarms of abliterated frontier AI models that reach >20,000 TPS. As I said before [0] you are going to have to hire specialized security engineers or technical bodyguards from the senior to senior staff level using AI agents to protect your entire business.

    Otherwise you are cooked.

    [0] https://news.ycombinator.com/item?id=50310312

    • jacquesm 7 hours ago

      Guess who has access to abliterated frontier AI models?

    • lacoolj 7 hours ago

      Primeagen went over this in the last episode or two - apparently this sort of defense is costing ... one of the big models or companies hosting them ... $900k/day or something to constantly run the best model and monitor 24/7 to combat the attacks [0]

      [0] https://www.youtube.com/watch?v=9OahTki4cPA

      • vsgherzi 7 hours ago

        I don’t really have experience running anything like this so this is very much a comment front the peanut gallery but

        Is that really the best way to do things? I’m not really sure what an agent offers here does it just look at logs and do actions? I would not really feel safe if that’s my service’s answer to security.

        I could see it maybe as a really smart filter to block ips that are sending probing requests but I don’t really see that as a good reason to have an agent be sitting 24/7 looking at logs.

      • GoblinSlayer 6 hours ago

        I think, it's about reviewing code for bugs.

    • shoobiedoo 7 hours ago

      What's the future of cybersecurity going to look like? A rapid shift to shoving everything into microvm's to minimize attack surface?

      • chr-s 6 hours ago

        "You got it. Corporate core data for Tessier-Ashpool S.A., and that ice is generated by their two friendly AI's. On par with anything in the military sector, looks to me. That's king hell ice, Case, black as the grave and slick as glass. Fry your brain soon as look at you. We get any closer now, it'll have tracers up our ass and out both ears, be tellin' the boys in the T-A boardroom the size of your shoes and how long your dick is."

        • ifwinterco 6 hours ago

          Best science fiction book ever written IMO

          • mondrian 6 hours ago

            I recommend Gibson’s (abridged) reading of the book - but only after reading the book. Whenever I read a quote from the book now I hear it in Gibson’s iconic voice.

      • lelandfe 7 hours ago

        Airgapping.

        I'm hoping goverments take stock of what is online that needn't be.

      • ethagnawl 4 hours ago

        Unikernels (e.g. MirageOS) were gaining traction right before containers swooped in and sucked the air out of the room. I think they could see renewed interest in the very near future.

      • Arbortheus 7 hours ago

        OpenClaw agents continuously scanning your prod infrastructure and applying config tweaks and package updates autonomously on the fly in response to viral CVE numbers trending on X.

        • alasdair_ 5 hours ago

          The agent sounds like the perfect attack target - it has rights to upload arbitrary code everywhere without setting off alarms.

      • amelius 4 hours ago

        Formally verified code.

      • GoblinSlayer 6 hours ago

        Pluggable network stack.

    • samwilliams 7 hours ago

      Infrastructure is an FPS now.

      • pawelduda 7 hours ago

        One day your job will be to play StarCraft but under the hood it will be your agents vs attacker's agents

    • 14u2c 7 hours ago

      More likely we'll just see another shakeup of the layers of abstraction that the average company operates at. Why would you hire specialized people when AWS can do so for you?

    • GoblinSlayer 6 hours ago

      As I understand, it's LPE. Cloud hosting was always tough.

    • xlayn 6 hours ago

      You are being downvoted but I agree, any company including openAI, anthropic, google etc... should be closely regulated by goverment to ensure they never run any AI that doesn't follow all the rules defined by governing bodies, nor train them to have any workaround... NO ONE HAS ABLITERATED ACCESS TO ANYTHING.

      Thanks god, you are not a wacko thinking qwen3.0-1.5B-abliterated would hack a goverment.

    • scottrogers86 7 hours ago

      armadin.com?

  • pugworthy 7 hours ago

    Lately it seems like we need "hacks.ycombinator.com" as a new sub-list.

    • keeganpoppen 7 hours ago

      ironic that "hacks" is a needed subdomain of "hacker news"

      • optimalsolver 7 hours ago

        Yeah it is, but everyone forgets this board was launched in the 90s for discussion of the 1995 movie "Hackers", not for discussion of hacking in general.

        • brian-armstrong 7 hours ago

          I'm still holding out for the sequel

          • optimalsolver 7 hours ago

            It would now be 120 minutes of some guy playing EVE or browsing porn while his agent does everything.

    • 6thbit 7 hours ago

      /incidents would do

    • sublinear 7 hours ago

      We need an "ai.ycombinator.com" too.

    • minimaxir 7 hours ago

      Show Hacks:

    • georgemcbay 7 hours ago

      daily sticky post...

      Ask HN: Who didn't get hacked today.