NSA Internal Network Down

(reuters.com)

109 points | by nodoodles 2 hours ago ago

50 comments

  • lucascdotnet an hour ago

    I wonder (and have no idea) whether the increased reports of services being down is a case of the Baader–Meinhof phenomenon, or a actual increase in failure rate.

    • btown 17 minutes ago

      If I was asked to armchair-pattern-match, I'd say that there's a combination of two things: (a) a wider fear-based environment where rapid modernization is being pushed heavily, and (b) an increase in tools that give false confidence on change management.

      AI is tuned to speak confidently and ship independently; it is rare that it will suggest that one go back to a broad survey of stakeholders to ensure no Chesterton's Fence is being torn down (https://fs.blog/chestertons-fence/). "I've read the documentation and logs, and nothing suggests this system needs these legacy capabilities/components" is something we hear all day, and it is a siren's song if there ever was one.

      Couple that with a general (and valid!) consensus that modernization of legacy systems must be accelerated for a variety of reasons - to mitigate this new and continuing "Black October" of hyper-agile agent-driven cyberattacks (whether the network is public or private); to have more flexible infrastructure for new features to stay ahead; to allow human teams to show their efficacy in a world where technology jobs are increasingly at risk - and you have a perfect storm where change management is no longer what it once was.

    • Someone1234 15 minutes ago

      That's my impression; a total of zero of the popular recent ones have been related to "AI" or Cyber attacks, but every comment in every thread, talks about those two as if they were the underlying cause.

      The FAA's recent outage for example, at its core, seems like people trying to rush needed upgrades that were slow for a specific reason (e.g. reducing the testing window for 90 to 30-days).

      The Microsoft outage seems like three different things linked together (and poor "full failure" recovery prep, lengthening the outage).

      The bank "outage" may not have even existed, 2/3 of the banks claimed they didn't have an outage at all, and the other said it was a limited technical issue. The articles were based on DownDetector spikes.

    • kesslern 15 minutes ago

      This situtation reminds me of the East Palestine train derailment in Ohio. A bad train accident caused a big chemical spill, made national news. For months afterward, any issue with any train in the state was reported as news. People thought the trains were breaking down. In reality, train derailments happen pretty often but it's not usually a big deal so nobody hears about it.

    • NitpickLawyer 39 minutes ago

      100%. Everything is rage-bait nowadays. And AI is the newest toy they can use. Every time we've had something go down in the past ~6months, "AI did it" was all over the comments.

      • rsoto2 30 minutes ago

        https://www.optimizely.com/field-notes/articles/how-we-use-a...

        https://www.bleepingcomputer.com/news/security/optimizely-su...

        Is it possible that the false of productivity is leading to worse products and massive loss in valuable headcount? Instead of AIs directly writing bad buggy code? Or is "everything rage-bait, and AI is all sunshine"

        • ragequittah 9 minutes ago

          I believe it's also possible that a large portion of the world wants this to be true more than it actually is so attribution to things that have nothing to do with AI go directly to AI as a knee-jerk reaction.

          • GolfPopper 5 minutes ago

            I'm not a fan of where I see LLMs taking us, but they're just the latest notch in decades-long institutional decay.

        • cochne 26 minutes ago

          These kinds of things happened before AI. It's not clear whether the incidence has increased since AI, or maybe AI is leading to faster development and therefore higher rates of incidents may be due to shipping more code, faster, while perhaps maintaining a similar or better level of incident to code/progress ratio. Or, there could just be more attention as GP suggests. Pointing to one example of a company that's using AI and has has a major incident doesn't really tell us much, especially when most companies who haven't had major incidents are probably using AI in similar amounts.

          It would be interesting if someone could do a proper analysis, though!

      • matt3210 26 minutes ago

        what was your system prompt again?

    • EvanAnderson 29 minutes ago

      Aside: I forget the name of the Baader–Meinhof phenomenon when it hasn't been discussed in awhile. When I'm reminded of it I begin seeing it happening frequently for the next few days. I wish I was joking. I assume it's the Baader–Meinhof phenomenon phenomenon.

    • dmix an hour ago

      The media will certainly try to give that impression either way. They serve the doomscrollers now.

      • causal 24 minutes ago

        Yeah "cyber thing went down" has certainly become a reporting trend. I'm not sure we would have even heard about NSA internal network issues a year ago. May or may not be significant.

  • tgsovlerkhgsel an hour ago

    That's a LOT of things going down, and I haven't heard of some major infrastructure outage likely to affect many sites in this manner.

    Coincidence or is there some common explanation that I missed?

    • postsantum 2 minutes ago

      Appstore connect is also down

    • bhouston an hour ago

      There is a lot of things going down.

      I think the opportunity right now where we have powerful AI hacking systems but not enough protections is to establish long-term presence in all networks of interest. Not bring them down.

      • colinrand an hour ago

        Completely agree. Things are moving too fast and so are breaking.

        For example, we updated a GCP managed base image of Ubuntu that was released with 350 CVEs patched, some were as high as 9.8. In doing so, they (GCP) changed the DEFAULT setting for hardware GRO from off by default to on. That's a massive networking change packed into a security update.

        We only caught it AFTER it started causing issues in production. We didn't have the OS pinned because (decently valid reasons) or catch it in non-prod (decent reason + human mistake).

        This is on a modern highly automated tested resilient well architected platform.

        I can only imagine these other platforms are dramatically more exposed that we are, hence these common large scale outages.

      • pianopatrick 9 minutes ago

        That assumes that people did not already have long-term presence in networks of interest.

        More plausible to me is that spy agencies had previously found and used some of these hundreds or thousands of vulnerabilities to hack into those networks just fine. But now other people like cyber criminals are using these vulnerabilities via AI. That is messing up both the attackers who had already gotten in quietly and do not want the attention and the defenders who now have to respond to a noisy intrusion they cannot ignore.

      • daheza an hour ago

        None of these have been due to hacking? It has all been just standard issues probably increased by cost cutting measures.

        • bhouston an hour ago

          Or the increased use of AI agent coding.

    • chabes an hour ago

      What do people expect when top government offices are occupied by former Fox News hosts?

  • gnabgib an hour ago

    CNN Source (27 points, 5 comments) https://news.ycombinator.com/item?id=50532849

  • ragle 8 minutes ago

    I have also been curious about frequency bias / Baader–Meinhof - but I am also wondering: what are the chances that a state actor is responsible for the last week of issues?

    Trump is headed to China for a meeting with Xi and Putin.

    We have 3 carriers off the coast of Iran and (despite Trump's pleas) NATO is essentially pushing their boot into Putin's neck w/ continued Ukrainian strikes on Russian oil infra.

    I've seen a few takes from L3 Harris / FAA employees calling the official line about the ATC issue bunk and there are 3 other major issues in the last week, now this NSA outage - curious if these incidents could be the BRICS axis flexing their own cybermuscles (independently in multiple separate attacks with carefully placed assets) ahead of their meeting with Trump in China?

    An interesting geopolitical Mexican standoff, if true.

    I'm not claiming this is true. I'm literally pulling it out of my rear with little context beyond a bunch of random news stories and X posts, it's pure pedestrian speculation / curiosity.

    Would love someone steeped in national security / geopolitics (like: professionally as their job, not armchair analysts) to offer their take on this.

    Is this a dumb, naive theory?

  • bayat 20 minutes ago

    Following the series of such events over the past couple of days, I have to say I’m a bit surprised by how quickly some in this community are brushing them off as AI-related pain or simply another example of the clickbait-driven media ecosystem.

    I understand the skepticism, and I certainly don’t think we should go down a rabbit hole of conspiracy theories. But shouldn’t we also be a little skeptical of dismissing these events so quickly? Is it unreasonable to at least consider the possibility that they’re not entirely coincidental, or to wonder why so many seem to be taking place in the US?

    • arrowleaf 5 minutes ago

      IMO it's unreasonable not to think they are all related in some way, even if that relation is just bad actors seeing what other bad actors have been capable of recently. This all started with the freezers in Europe and seems to be snowballing since.

  • swalsh 15 minutes ago

    I have this distinct memory in I think Jan 2020 where I looked at the front page of CNN or NYT or something. Big picture of Trump impeachment, and proceedings. Small picture at the bottom of the page of China dealing with this virus. Worthy news, but not top page news. Nobody was paying attention. Feb, more people started paying attention, but it wasn't top page yet. By March, people were starting to realize that this is a serious crisis that's going to severely impact their lives.

    I'm getting Feb 2020 vibes. I don't know what's going on, but this feels like more than just some routine outages going on.

    • sanex 6 minutes ago

      Well the good news is that even though 2020 and 2021 were weird years we're still here. Humanity is resilient, we always figure it out.

    • ssl-3 11 minutes ago

      I was sicker than I'd ever been for ~all of February of 2020.

      So far this fall, none of my systems have gone haywire (yet!), so I forecast that we'll be OK.

  • ericcholis an hour ago

    There's a part of me that doesn't want to know this....

  • howmayiannoyyou 17 minutes ago

    https://chinaselectcommittee.house.gov/media/reports/strange...

    Allowing our infrastructure to be manufactured elsewhere, maintained by companies who struggle to retain top talent, and overseen by folks out of touch with cutting edge threats, and funded on annual budget cycles by legislators even more out-of-touch ... is a seemingly insurmountable set of obstacles.

  • kinabalu76 11 minutes ago

    I see the shoe is now on the other foot NSA.

  • joshka 44 minutes ago

    > The NSA did not immediately return a message seeking comment about the reported outage.

    Did you try sending it via outlook.com?

    • Semiapies 9 minutes ago

      Just whisper the question to your phone.

  • throw7 26 minutes ago

    Memo to the NSA... pulling the plug won't work. ;)

  • swozey 2 hours ago

    Just in time for the FEMA national emergency test this week

  • stevenhubertron an hour ago

    It’s been a hella few weeks. Hard to believe it’s coincidence even if you want to blame AI.

    • rglover 44 minutes ago

      It's AI. Or more specifically: rushed, careless usage of AI under the pretense that it's already "solved" software and you can haphazardly push junk to the world and expect real systems to run on it.

      • londons_explore 18 minutes ago

        Traditional software development normally involves at least a few veterans on the team familiar with almost every line of code.

        However, AI works like a speedy intern who hasn't yet got to grips with the whole codebase, and sometimes makes assumptions. That's mostly because the codebase is usually bigger than the context window, so it is 'writing blind'.

      • ranger_danger 16 minutes ago

        Do you have any sources for this claim?

  • reedf1 2 hours ago

    Anyone have knowledge of the backend of this system?

    • aeonik an hour ago

      Are you implying that you DON'T have detailed docs and architecture of one of the most classified and secretive organizations in the world? ;-P

      • infinite_spin an hour ago

        php and spit

        • grep_it 29 minutes ago

          ... and Doug's perl script, which was never committed to source control. It lives on a single box and silently upholds a number of implicit system invariants. In someones inbox is an email from Doug, sent just before he retired, that the script needs to be copied over to the new box before the hardware migration is complete.

      • doublerabbit an hour ago

        You telling me you don't? Bro just download it off Limewire.

        • SketchySeaBeast 30 minutes ago

          We might have more luck checking the War Thunder Forums.

          • zmgsabst 4 minutes ago

            Wireporn on Discord can have some spicy server/rack labels.

    • Havoc 7 minutes ago

      My money is on something absolutely antique but wrapped in modern post quantum encryption.

      Not unlike Bloomberg terminals…dos era UI but does the job well

    • bix6 21 minutes ago

      Just join Hegseth’s signal chat and he’ll give you the deets. Ideally pick a username like WAREAGLE.

    • Zambyte 20 minutes ago

      Surely

  • vrganj an hour ago

    There's some speculation that all of these recent outages might be related to attempts at taking Genesis Mission online - the Trump / Oracle project to have AI tie into government and private services:

    https://en.wikipedia.org/wiki/Genesis_Mission

    https://substack.com/@thedreydossier/note/c-389611426