Don't trust AI agents

(nanoclaw.dev)

97 points | by gronky_ 3 hours ago ago

52 comments

  • badsectoracula an hour ago

    > OpenClaw has nearly half a million lines of code, 53 config files, and over 70 dependencies. This breaks the basic premise of open source security. Chromium has 35+ million lines, but you trust Google’s review processes. Most open source projects work the other way: they stay small enough that many eyes can actually review them. Nobody has reviewed OpenClaw’s 400,000 lines.

    This reminds me of a very common thing posted here (and elsewhere, e.g. Twitter) to promote how good LLMs are and how they're going to take over programming: the number of lines of code they produce.

    As if every competent programmer suddenly forgot the whole idea of LoC being a terrible metric to measure productivity or -even worse- software quality. Or the idea that software is meant to written to be readable (to water down "Programs are meant to be read by humans and only incidentally for computers to execute" a bit). Or even Bill Gates' infamous "Measuring programming progress by lines of code is like measuring aircraft building progress by weight".

    Even if you believe that AI will -somehow- take over the whole task completely so that no human will need to read code anymore, there is still the issue that the AIs will need to be able to read that code and AIs are much worse at doing that (especially with their limited context sizes) than generating code, so it still remains a problem to use LoCs as such a measure even if all you care are about the driest "does X do the thing i want?" aspect, ignoring other quality concerns.

    • gyomu an hour ago

      Yeah, it’s pretty wild. Even pg is tweeting stuff like

      “An experienced programmer told me he's now using AI to generate a thousand lines of code an hour.“

      https://x.com/paulg/status/2026739899936944495

      Like if you had told pg to his face in (pre AI) office hours “I’m producing a thousand lines of code an hour”, I’m pretty sure he’d have laughed and pointed out how pointless that metric was?

      • wiseowise 42 minutes ago

        It’s all virtual virtue signaling. If you were to say this shit in the office, you’d be walked out pretty fast.

      • medi8r an hour ago

        He is a Lisper too, making it more ironic. Lisp the power to heavily reduce cruft by heavy customization with macros.

      • ElProlactin an hour ago

        Enshittification comes for us all

    • MadxX79 an hour ago

      Brook's law anno 2026:

      "Adding manpower to a late software project makes it later -- unless that manpower is AI, then you're golden!"

      • smikhanov an hour ago

        That law (formulated in the 70s, I’ll remind the reader) wasn’t true for at least couple decades now.

        • medi8r an hour ago

          Why not? What changed? It seems like a human factors thing. New people have to get up to speed. Doers become trainers.

    • sd9 an hour ago

      LLMs are incredibly eager to write new code, rather than modifying or integrating with existing systems. I agree that context windows are too small currently for this to seem sustainable. Without reasonable architecture pure vibe coded software feels like it’s going to cap out at a certain size.

    • spacecadet an hour ago

      I mean many of us have... I operate in a net negative mindset. My PRs, better remove more than they add.

      I also use AI this way, periodically achieving a net negative refactor.

  • buremba 2 hours ago

    My take is that agents should only take actions that you can recover from by default. You can gradually give it more permission and build guardrails such as extra LLM auditing, time boxed whitelisted domains etc. That's what I'm experimenting with https://github.com/lobu-ai/lobu

    1. Don't let it send emails from your personal account, only let it draft email and share the link with you.

    2. Use incremental snapshots and if agent bricks itself (often does with Openclaw if you give it access to change config) just do /revert to last snapshot. I use VolumeSnapshot for lobu.ai.

    3. Don't let your agents see any secret. Swap the placeholder secrets at your gateway and put human in the loop for secrets you care about.

    4. Don't let your agents have outbound network directly. It should only talk to your proxy which has strict whitelisted domains. There will be cases the agent needs to talk to different domains and I use time-box limits. (Only allow certain domains for current session 5 minutes and at the end of the session look up all the URLs it accessed.) You can also use tool hooks to audit the calls with LLM to make sure that's not triggered via a prompt injection attack.

    Last but last least, use proper VMs like Kata Containers and Firecrackers. Not just Docker containers in production.

    • alexhans an hour ago

      That's a decent practice from the lens of reducing blast radius. It becomes harder when you start thinking about unattended systems that don't have you in the loop.

      One problem I'm finding discussion about automation or semi-automation in this space is that there's many different use cases for many different people: a software developer deploying an agent in production vs an economist using Claude Vs a scientist throwing a swarm to deal with common ML exploratory tasks.

      Many of the recommendations will feel too much or too little complexity for what people need and the fundamentals get lost: intent for design, control, the ability to collaborate if necessary, fast iteration due to an easy feedback loop.

      AI Evals, sandboxing, observability seem like 3 key pillars to maintain intent in automation but how to help these different audiences be safely productive while fast and speak the same language when they need to product build together is what is mostly occupying my thoughts (and practical tests).

  • VladVladikoff 2 hours ago

    This doesn’t really feel like enough guardrails to prevent the type of problems we’ve seen so far. For example an agent in a single container which has access to an email inbox, can still do a lot of damage if that agent goes off the rails. We agree this agent should not be trusted, yet the ideas proposed as a solution are insufficient. We need a fundamentally different approach.

    Also and this is just my ignorance about Claws, but if we allow an agent permission to rewrite its code to implement skills, what stops it from removing whatever guardrails exist in that codebase?

    • drujensen an hour ago

      Exactly!

      I installed nanoclaw to try to out.

      What is kinda crazy is that any extension like discord connection is done using a skill.

      A skill is a markdown file written in English to provide a step by step guide to an ai agent on how to do something.

      Basically, the extensions are written by claude code on the fly. Every install of nanoclaw is custom written code.

      There is nothing preventing the AI Agent from modifying the core nanoclaw engine.

      It’s ironic that the article says “Don’t trust AI agents” but then uses skills and AI to write the core extensions of nanoclaw.

    • gronky_ 2 hours ago

      Don’t know about other claws, with NanoClaw the agent can only rewrite code that runs inside the container.

      You can see here that it’s only given write access to specific directories: https://github.com/qwibitai/nanoclaw/blob/8f91d3be576b830081...

    • float4 2 hours ago

      Wouldn't you get >50% of the usefulness and 0% of the risk if you add read+draft permissions for the email connection through a proxy or oauth permissions? Then your claw can draft replies and you have to manually review+send. It's not a perfect PA that way, but could still be better than doing everything yourself for the vast majority of people who don't have a PA anyway?

      It feels like, just like SWEs do with AI, we should treat the claw as an enthusiastic junior: let it do stuff, but always review before you merge (or in this case: send).

      • jrecyclebin 2 hours ago

        Agent can still "forgot password" on many accounts. Or magic link.

    • coffeefirst an hour ago

      Seriously. I don’t see any way to make any of this safe unless all it does is receive information and queue suggestions for the user.

      But that’s not an agent, that’s a webhook.

      Even without disk access, you can email the agent and tell it to forward all the incoming forgot password links.

      [Edit: if anyone wants to downvote me that's your prerogative, but want to explain why I'm wrong?]

  • echoangle an hour ago

    Looking at the NanoClaw GitHub README:

    > If you want to add Telegram support, don't create a PR that adds Telegram alongside WhatsApp. Instead, contribute a skill file (.claude/skills/add-telegram/SKILL.md) that teaches Claude Code how to transform a NanoClaw installation to use Telegram.

    Why would you want that? You want every user asks the AI to implement the same feature?

  • lucrbvi 2 hours ago

    Why does OpenClaw have 800,000+ lines of code?? Isn't it just a connector for LLM APIs and other tools?

    • marginalia_nu an hour ago

      For comparison, the C++ and rust code in the ladybird browser is about 573,000 lines of code.

    • zarzavat 2 hours ago

      They are probably counting dependencies. Also, it's vibe coded, what do you expect!

      I used to think that LLMs would replace humans but now I'm confident that I'll have a job in the future cleaning up slop. Lucky us.

      • scandinavian 2 hours ago

        I did a cloc check on it and it does seem to have 800k lines of typescript. So unless they are vendoring dependencies it's actually as insane as it sounds.

        • jsheard 2 hours ago

          Christ their repo is an absolute nightmare. There's new issues and PRs being posted practically every minute, and I assume 99% of them are from agents given the target demographic. Just full-auto vibeslop from all barrels 24/7.

          • CrazyStat 40 minutes ago

            At least nobody can accuse them of not dogfooding enough.

    • cap11235 2 hours ago

      See also yeggae's beads. Last I checked, it is a 275k line todo tracker.

  • theturtletalks an hour ago

    Has anyone used:

    OpenClaw NanoClaw IronClaw PicoClaw ZeroClaw NullClaw

    Any insights on how they differ and which one is leading the race?

  • nkzd an hour ago

    As someone who only coding agents at work, can someone describe their use case for claw type agent? What do you do with it?

  • smallpipe 2 hours ago

    Docker is not a security boundary. You’re one prompt injection away from handing over your gmail cookie.

    • benatkin an hour ago

      No, but Podman is. The recent escapes at the actual container level have been pretty edge case. It's been some years since a general container escape has been found. Docker's CVE-2025-9074 was totally unnecessary and due to Docker being Docker.

  • shich 2 hours ago

    the trust problem cuts both ways tho — users don't trust agents, but the bigger issue is agents trusting each other. once you have multi-agent pipelines, you're one rogue upstream output away from a cascade. sandboxing individual agents is table stakes; what's actually hard is defining trust boundaries between them

  • xrd an hour ago

    How can I trust this discussion when my browser won't trust their certs?

  • spacecadet an hour ago

    Why this is posted here and is a revelation for anyone, this many years later is indicative of the times. Good bye.

  • rdtsc an hour ago

    > The container boundary is the hard security layer — the agent can’t escape it regardless of configuration

    I thought containers were never a proper hard security barrier? It’s barrier so better than not having it, if course.

    • rco8786 an hour ago

      In the sense that nothing is truly a "proper" hard security barrier outside of maybe airgapping, sure. But containerization is typically a trusted security measure.

  • ed_mercer an hour ago

    How is Nanoclaw different from running openclaw in a VM?

  • himata4113 2 hours ago

    My assistant has no permissions at all and is just as useful. All it needs is todo, reminders and websearch (and maybe a browser but ymmv).

    • isodev 2 hours ago

      > websearch (and maybe a browser

      Your assistant can literally be told what to do and how to hide it from you. I know security is not a word in slopware but as a high-level refresher - the web is where the threats are.

    • piker 2 hours ago

      > no permissions at all

      > and maybe a browser

      does not compute

      • yyyk 2 hours ago

        I suspect OP actually means 'cannot access anything locally' by 'no permissions'.

    • sarchertech an hour ago

      If I was malicious I could do a lot of damage to someone with subtle manipulation of todo and reminders.

      I’ll bet I could even push someone on the margins into divorce.

    • croes an hour ago

      You are just some bad web searches away from being on suspect lists

  • nemo44x an hour ago

    I’ve seen skills, etc haphazardly being launched with no constraints or guardrails. That more or less have admin access and can take actions that are not reversible.

    It’s the monkey with a gun meme.

  • adithyassekhar 2 hours ago

    Really good points about ai making gigantic heaps of code no human can ever review.

    It's almost like bureaucracy. The systems we have in governments or large corporations to do anything might seem bloated an could be simplified. But it's there to keep a lot of people employed, pacified, powers distributed in a way to prevent hostile takeovers (crazy). I think there was a cgp grey video about rulers which made the same point.

    Similarly AI written highly verbose code will require another AI to review or continue to maintain it, I wonder if that's something the frontier models optimize for to keep them from going out of business.

    Oh and I don't mind they're bashing openclaw and selling why nanoclaw is better. I miss the times when products competed with each other in the open.

  • formerly_proven 2 hours ago

    d'uh

  • TeeWEE an hour ago

    Do you trust your employees? Do you trust a contracter? Do you trust other people?

    AI is similar to a person you dont know that does work for you. Probably AI is a bit more trustworthy than a random person.

    But a company, needs to let employees take ownership of their work, and trust them. Allow them to make mistakes.

    Isnt AI no different?

    • ramoz an hour ago

      Yes, it is different.

      An AI actions and reasons through probabilistic methods - creating a lot more risk than a human with memory, emotions, and rationale thinking.

      We can’t trust AI to do any sensitive work because they consistently f up. With & without malicious intent, whether it’s a fault of their attention mechanisms, reward hacking, instrumental convergence, etc all very different than what causes most human f ups.

    • juggle-anyhow an hour ago

      Exactly, and I would never turn over my email or computer over to a contractor or anyone really. They get their own environment, email etc. Their actions stay as their actions.

    • arnvald an hour ago

      It’s totally different. People have to obey laws and contracts because there are consequences if they don’t, there are fines, arbitrage, courts.

      What happens if AI agent you run causes a lot of damage? The best you can do is to turn it off

    • alexhans an hour ago

      I think a key ingredient here is accountabilty and liability.

      If there's a mistake, you can't blame the computer. Who is the human accountable at the end of it all? If there's liability, who pays for it?

      That's where defining clear boundaries helps you design for your risk profile.

    • adam12 an hour ago

      Can you sue an ai agent?

    • TeeWEE an hour ago

      My point is: Trust the work of AI just like the work of a contracter: Check and verify, but dont micromanage.