I kind of have a different idea of agents. I totally believe in a deterministic scaffold but I really think that an agent should be as deterministic as possible - the more code, the better.
Think of a typical loop we may ask of Claude Code today (assume we are not using TDD): run some test suite with fail fast mode, diagnose if the failure is due to recent feature changes (pass reference to backend/frontend, github issues, PRD,...). Ask CC to decide if test failed due to feature change and then update the test. Perhaps ask CC to use sub-agent to investigate and fix (if deemed so). Commit each fix, move on to next.
I know, this has so many ways to make blunder but I am talking about the agent here, not our error-prone test maintenance. What if we had an agent that had context of your codebase, deterministically ran test suite, linter, hooks, etc. The "English" prompt would become a code loop with the LLM only brought in to decide if a test has failed because of feature change. Also, we can extract git log, JIRA and what not.
Each tool here is real code. Executable code that calls others and only prompts when they meet edge cases. Edge cases are defined but we can now accelerate the maintenance of these tools using agents themselves. But the system is built on "programs that do one thing and do it well" and then reach out to an LLM for its specific edge case. The agent is how these executables work with each other.
I’m definitely on the deterministic code train as well. All of my success for long running tasks has been around wrapping the agentic harness (cc, codex-cli, etc.) in a deterministic workflow with deterministic gates. We need a name for this outer layer. In my mind that is the true harness because it constrains the agents failure mode. I think flow engineering has been proposed. Maybe it’s the agentic exoskeleton?
100% agree that the more deterministic code the better up to the limit where you need the LLM's ability to be non-deterministic to kick in.
There is this ACM blog post called "Manual Work is a Bug" [0] that was originally written to help humans automate processes using code. I find it just as applicable today as when it was written. You and the LLM look at what has to be done and then figure out the scripts/tools to make it happen. You then tie those tools into a system.
The more I use the above the more it makes sense and the worse the whole "just commit the prompt" seems like nonsense.
Awesome work! This is really impressive. I gave a GitHub star.
I build precision-editing tools for AI coding agents (hic-ai.com) and worked out thousands of JSON-wrangling and regex issues, so I can verify they are indeed a bit of a pain, across all possible failure modes that AI coding agents and models and harnesses can produce. Anyway, I completely agreed with everything in your article, though I would suggest however that agents need *three* things at runtime to fix a defect: great logging and a clear error response (just like you have it), but also, precision-editing tools that enable agents to make the minimal, surgical change without touching or copying any other portion of the file. These actually change not just the feedback but also the options available to the agent and capabilities in the midst of the workflow to self-heal. If Ambiance adds a kernel to buffer the LLM from the outside world, HIC Mouse adds a "kernel" or buffer between the LLM and its own environment and file system. Anyway, this is such a cool project. Please reach out if you ever add MCP support for Ambiance -- I'm happy to release a new version of Mouse that supports it. Again, great work.
I really like this idea and the way you mapped the concepts to unix primitives. Indeed llms are already "unix native". I've been experimenting with similar event driven workflows using k8s primitives but that's one level up the stack. This makes a whole lot of sense to me in terms of organizing a shared mental model. Will definitely check it out. Thanks for the good work.
Actually I was initially going to put this on my Arch machine because utilizing FUSE made a lot of sense to me, but I realized that I use my mac a lot more often so I suppose that's what caused that choice.
What has been the most helpful when developing harnesses:
> When in doubt, simplify. Remove, trim and minimize. Reproduce issues in as small cases as possible, understand the full design completely, there is no shortcuts for this.
I'm kind of in the same boat and it's been pestering me for months. Every agent is simply a less capable Claude Code.
If it had a lossless, massive context window (100m-1b tokens), then it will squash everything. Give it bash + r/w and it can in theory /goal anything.
I think there's something to be gained in a production environment be siloing agents for reproducebility/auditability, but I suspect that will go away in the future.
There's that video of a silly demo someone made of an OS that was just nested copilot instances that generated the HTML of each window, which allowed you to do whatever you could imagine. It was seen as silly because it was, but that seems truly transformative.
i think the point is less about what agent you should or should not do and more about what is the natural harness for an agent to succeed in. And agents are often autonomously doing things right now, why would you want claude code doing such stuff for you?
How much do the labs post-train on the harness inputs & outputs? That's a critical piece to understand if a "generic" harness is at all possible
I kind of have a different idea of agents. I totally believe in a deterministic scaffold but I really think that an agent should be as deterministic as possible - the more code, the better.
Think of a typical loop we may ask of Claude Code today (assume we are not using TDD): run some test suite with fail fast mode, diagnose if the failure is due to recent feature changes (pass reference to backend/frontend, github issues, PRD,...). Ask CC to decide if test failed due to feature change and then update the test. Perhaps ask CC to use sub-agent to investigate and fix (if deemed so). Commit each fix, move on to next.
I know, this has so many ways to make blunder but I am talking about the agent here, not our error-prone test maintenance. What if we had an agent that had context of your codebase, deterministically ran test suite, linter, hooks, etc. The "English" prompt would become a code loop with the LLM only brought in to decide if a test has failed because of feature change. Also, we can extract git log, JIRA and what not.
Each tool here is real code. Executable code that calls others and only prompts when they meet edge cases. Edge cases are defined but we can now accelerate the maintenance of these tools using agents themselves. But the system is built on "programs that do one thing and do it well" and then reach out to an LLM for its specific edge case. The agent is how these executables work with each other.
I’m definitely on the deterministic code train as well. All of my success for long running tasks has been around wrapping the agentic harness (cc, codex-cli, etc.) in a deterministic workflow with deterministic gates. We need a name for this outer layer. In my mind that is the true harness because it constrains the agents failure mode. I think flow engineering has been proposed. Maybe it’s the agentic exoskeleton?
100% agree that the more deterministic code the better up to the limit where you need the LLM's ability to be non-deterministic to kick in.
There is this ACM blog post called "Manual Work is a Bug" [0] that was originally written to help humans automate processes using code. I find it just as applicable today as when it was written. You and the LLM look at what has to be done and then figure out the scripts/tools to make it happen. You then tie those tools into a system.
The more I use the above the more it makes sense and the worse the whole "just commit the prompt" seems like nonsense.
0 - https://queue.acm.org/detail.cfm?id=3197520
I know it's a type of blasphemy here, but deterministic workflows such as what you describe is where langgraph really shines imo.
Awesome work! This is really impressive. I gave a GitHub star.
I build precision-editing tools for AI coding agents (hic-ai.com) and worked out thousands of JSON-wrangling and regex issues, so I can verify they are indeed a bit of a pain, across all possible failure modes that AI coding agents and models and harnesses can produce. Anyway, I completely agreed with everything in your article, though I would suggest however that agents need *three* things at runtime to fix a defect: great logging and a clear error response (just like you have it), but also, precision-editing tools that enable agents to make the minimal, surgical change without touching or copying any other portion of the file. These actually change not just the feedback but also the options available to the agent and capabilities in the midst of the workflow to self-heal. If Ambiance adds a kernel to buffer the LLM from the outside world, HIC Mouse adds a "kernel" or buffer between the LLM and its own environment and file system. Anyway, this is such a cool project. Please reach out if you ever add MCP support for Ambiance -- I'm happy to release a new version of Mouse that supports it. Again, great work.
> Awesome work! This is really impressive. I gave a GitHub star.
{ aislop pitch}
> Again, great work.
i can bet you didnt actually read the op. i hate these comments so much. selfish and rude.
I did indeed read every word. And read the code. Don't be rude.
What's with this "harness" word people have been trying to adopt lately? Are we all going rock climbing?
I really like this idea and the way you mapped the concepts to unix primitives. Indeed llms are already "unix native". I've been experimenting with similar event driven workflows using k8s primitives but that's one level up the stack. This makes a whole lot of sense to me in terms of organizing a shared mental model. Will definitely check it out. Thanks for the good work.
thanks a lot! let me know what you think -- i'd love to share ideas with people doing similar stuff!
I do have a gripe already... The readme says macos, however as of yet I don't see the actual dependency. Sort of defeats the *nix spirit as well, no?
Actually I was initially going to put this on my Arch machine because utilizing FUSE made a lot of sense to me, but I realized that I use my mac a lot more often so I suppose that's what caused that choice.
Yeah, a little... I was mostly building this for myself so I didn't really think about other platforms but I'll get to it asap :)
good news. now you have a feature request.
What has been the most helpful when developing harnesses:
> When in doubt, simplify. Remove, trim and minimize. Reproduce issues in as small cases as possible, understand the full design completely, there is no shortcuts for this.
Why are we not just using Claude Code or Codex on our machine and using this thing? Real question...
I'm kind of in the same boat and it's been pestering me for months. Every agent is simply a less capable Claude Code.
If it had a lossless, massive context window (100m-1b tokens), then it will squash everything. Give it bash + r/w and it can in theory /goal anything.
I think there's something to be gained in a production environment be siloing agents for reproducebility/auditability, but I suspect that will go away in the future.
There's that video of a silly demo someone made of an OS that was just nested copilot instances that generated the HTML of each window, which allowed you to do whatever you could imagine. It was seen as silly because it was, but that seems truly transformative.
i think the point is less about what agent you should or should not do and more about what is the natural harness for an agent to succeed in. And agents are often autonomously doing things right now, why would you want claude code doing such stuff for you?
This one is open source, and potentially better, since it responds to (file system) events instead of polling.