No benchmarks, no info on which models are used, ai generated video, just a signup page with nothing else.
Anyhow, this kinda reminds me of that quote about architecture: "We replaced our monolith with micro services so that every outage could be more like a murder mystery."
It currently exposes 907 stored rows across seven benchmark families, with prompts, outputs, grades, and cost records. More benchmarks are coming soon.
Echo does not disclose its per-request routing decision because that policy is the product. We can, however, publish some of the eligible open-weight model pool, version dates, aggregate allocation mix, and evaluation settings without exposing the request-level recipe.
I've been working on a similar project and I found that it's easy to replicate Fable results if you use saturated benchmarks.
In my project, I wasted a huge amount of time trying to improve GPQA Diamond results above ~93% range. I realized my mistake when Fable dropped and made no improvement on this benchmark vs. Opus.
It's basically trying to replicate OpenRouter, which works pretty well and has a lot of nice features to abstract away any single provider, such as failover, metering, autoswitching, etc. It's actually a really smart infrastructure abstraction.
I just wish this were solving an actual problem rather than being a fairly transparent attempt to say something approximating, "Hey VCs, OpenRouter just became a unicorn but I can basically vibe code it"
Calling it "Fable-level" feels intellectually lazy / dishonest, but then again, what do you expect when there's so much money on the table.
> Fable-level results at 1/3 the cost using open-weight models
But we get ~$2500/mo worth of Fable credits for $200/mo on Anthropic pan? I'm still confused why people (who don't have to use API billing) are chasing open weight models based on cost.
Enterprise plans pay API costs, they're only subsidizing individual accounts and that's only because they have to compete with open-source models.
When they are successful at making those illegal/inaccessible, both Anthropic and ChatGPT are going to rip the band-aid off and inference will only be sold to those that can afford it.
> When they are successful at making those illegal/inaccessible
This would be like trying to outlaw Linux or peer-to-peer file sharing. It's technically possible to write and pass a law, but it's basically impossible to enforce it.
Because that is a short term solution, it won’t be offered forever. Large organisations have to purchase credits at $/tokens. Eventually everyone else will too.
The point still stands. The chinese labs don't have super discounted plans, so if the price per task benchmarks[1] are correct, and we apply the discount, you'll actually be paying more by using cheaper chinese models and this technique.
You are right that the privacy wording was too broad. We are fixing it now so it states explicitly that Echo does not use customer prompts, files, chats, or outputs to train or fine-tune models. We are updating the matching Terms language at the same time.
Echo also starts users with free credit and does not require a credit card to try it. The current signup flow did not make that clear enough, so we are fixing that presentation too.
Intuitively, your savings depend heavily on how hard the tasks are in the first place. If you have a base rate where 99% of your tasks can be routed to a cheap model, yeah, you can save a ton by not using Fable for that.
I'm not an expert on this, but this sounds a lot like a larger-scale MoE (Mixture of Experts) type of architecture.
As I understand it, in an MoE model, you essentially have hundreds of smaller sub-models ("experts") that are good at different tasks, and for every generated token, a single "master" model chooses which ones are most relevant to participate, and you only activate them.
In MoE systems the routing decision is made per-token, not per prompt or task. It’s one of ML’s many confusing naming conventions.
Even more confusingly, there are older pre-LLM MoE systems which ensemble and pool the predictions from multiple sub-components. For example in a random forest you could take the majority vote of the decision trees or the average of their numerical predictions.
After that, we developed neural net architectures for predicting a single thing like whether the user will click on your ad. An MMoE is in the same family.
And so now we are at massive MoE networks for LLMs which have similarities with MMoE in that the “decision” is about the very next token to predict.
Thanks for checking the individual rows. HumanEval+ is one small code slice, not the whole basis for the launch claim. The public evaluator currently contains 907 rows across seven benchmark families, and matched SWE-bench Verified and BigCodeBench runs are the next code evidence being added.
You also found a real UI bug: the inspector should show both stored answers and currently does not in some rows. We are fixing that.
On the row you reran: the page records a frozen matched run. It does not claim that Fable is incapable of solving that prompt on another run. We are adding repeated matched trials and making run count and variance explicit. Your rerun is exactly the kind of external check the row-level page is intended to make possible.
The broader result remains: Echo is competitive with Fable across the evaluated task mix at materially lower measured inference cost. We are filling in the harder agentic-code evidence now rather than asking anyone to infer it from HumanEval+.
I think approaches like this have potential. Only time will tell. This reminds me of the mixture of experts taken by deepseek r2 (I think it was r2, at least), but less specific models I guess.
I have often wondered how tools like GHCP choose the best model for the job when set to "auto".
This reminds me on OpenRouters report that combining multiple different models gave comparable performance to Fable 5. I think this approach has lots of potential. Maybe OpenAi was ahead of its time with GPT-5 (it being a router to different models rather than just being one new model)
You might be interested in a project that I'm working on, which is kind of like OpenRouter Fusion, but instead of waiting for all models to synthesize, we stream the best model immediately and background the rest. The background models then reconcile with an emoji reaction and optional reply. It gets similar results to Fusion and is a lot faster! There's a free version that leverages open weight models here: http://pellmell.ai.
I don’t find the recent spate of blog posts and systems delegating and combining LLMs to get better performance particularly interesting. Especially given that anyone who’s taken an ML 101 course has learned about ensemble methods.
While an LLM isn’t what you’d traditionally consider a weak learner, the theorems on learning systems clearly point to them being so in this context. The feigned surprise at combining them to yield better results seems disingenuous.
Even so, the work to predict which models are best suited for which task, how to delegate, and how to combine their outputs is interesting, especially if you’re placing a cost minimization objective on it. That said, this isn’t too far off from what many AI labs are already doing.
The entirety of "agents" and tool calls is a process of combining LLMs to get better results. Is it the same LLM in many cases? Yes. But it doesn't have to be.
It's the natural move that happened after people realized you couldn't throw away half a century of AI research.
Most of these focus on costs. But it is simply the case that the one-shot output did not scale for harder problems on workflows.
So this is the dogpile.com of the askjeeves, alta vista, and lycos approach? Time is a flat circle?
"Backed by YCombinator"
https://www.ycombinator.com/companies?query=tracerml
I don't see it?
haven't done yet the public yc launch, currently in s26
No benchmarks, no info on which models are used, ai generated video, just a signup page with nothing else.
Anyhow, this kinda reminds me of that quote about architecture: "We replaced our monolith with micro services so that every outage could be more like a murder mystery."
The evaluator is public here: https://echo.tracerml.ai/eval/
It currently exposes 907 stored rows across seven benchmark families, with prompts, outputs, grades, and cost records. More benchmarks are coming soon.
Echo does not disclose its per-request routing decision because that policy is the product. We can, however, publish some of the eligible open-weight model pool, version dates, aggregate allocation mix, and evaluation settings without exposing the request-level recipe.
New video is also being made.
Unrelated - but reminds me of my favorite quote by tenderlove:
"microservices turn function calls into distributed computing problems"
> No benchmarks, no info on which models are used, […]
The benchmarks are here → https://echo.tracerml.ai/eval/
They are not good benchmarks but at least they exist.
I've been working on a similar project and I found that it's easy to replicate Fable results if you use saturated benchmarks.
In my project, I wasted a huge amount of time trying to improve GPQA Diamond results above ~93% range. I realized my mistake when Fable dropped and made no improvement on this benchmark vs. Opus.
It's basically trying to replicate OpenRouter, which works pretty well and has a lot of nice features to abstract away any single provider, such as failover, metering, autoswitching, etc. It's actually a really smart infrastructure abstraction.
I just wish this were solving an actual problem rather than being a fairly transparent attempt to say something approximating, "Hey VCs, OpenRouter just became a unicorn but I can basically vibe code it"
Calling it "Fable-level" feels intellectually lazy / dishonest, but then again, what do you expect when there's so much money on the table.
OpenRouter's model router for coding isn't really sophisticated. This is very different.
It's easy enough to copy and paste in a prompt, no?
Eval tests while giving general indicators might not be similar for each use case.
> Fable-level results at 1/3 the cost using open-weight models
But we get ~$2500/mo worth of Fable credits for $200/mo on Anthropic pan? I'm still confused why people (who don't have to use API billing) are chasing open weight models based on cost.
Enterprise plans pay API costs, they're only subsidizing individual accounts and that's only because they have to compete with open-source models.
When they are successful at making those illegal/inaccessible, both Anthropic and ChatGPT are going to rip the band-aid off and inference will only be sold to those that can afford it.
The good ole American way.
> When they are successful at making those illegal/inaccessible
This would be like trying to outlaw Linux or peer-to-peer file sharing. It's technically possible to write and pass a law, but it's basically impossible to enforce it.
Enough to make it a non-started at the organizations that pay their bills. Everyone else isn't big enough to matter.
Because that is a short term solution, it won’t be offered forever. Large organisations have to purchase credits at $/tokens. Eventually everyone else will too.
This is what OpenAI and Anthropic are trying to make everyone believe. Most accountants will flinch at this (they already are).
The $200 odd plans are already out of reach of many, many people.
The attrition of customers if they were to get rid of these subscriptions plans would be untenable.
You may want to consider the incomes of developers outside the US, students, unemployed. $200/month is a lot to a lot of people.
The point still stands. The chinese labs don't have super discounted plans, so if the price per task benchmarks[1] are correct, and we apply the discount, you'll actually be paying more by using cheaper chinese models and this technique.
https://artificialanalysis.ai/agents/coding-agents#artificia...
They can still write code.
doesnt exist for enterprise plans
No single signin. Privacy policy allows training. No try it first without credit card. It's a good idea, but this looks premature.
You are right that the privacy wording was too broad. We are fixing it now so it states explicitly that Echo does not use customer prompts, files, chats, or outputs to train or fine-tune models. We are updating the matching Terms language at the same time.
Echo also starts users with free credit and does not require a credit card to try it. The current signup flow did not make that clear enough, so we are fixing that presentation too.
Thanks for calling both out.
If you want to try a similar idea that you can chat with immediately (free tier uses open weight models), I'm working on this: http://pellmell.ai
I assume trying without a credit card will bankrupt him immediately.
Intuitively, your savings depend heavily on how hard the tasks are in the first place. If you have a base rate where 99% of your tasks can be routed to a cheap model, yeah, you can save a ton by not using Fable for that.
So “1/3 the cost” really depends.
I'm not an expert on this, but this sounds a lot like a larger-scale MoE (Mixture of Experts) type of architecture.
As I understand it, in an MoE model, you essentially have hundreds of smaller sub-models ("experts") that are good at different tasks, and for every generated token, a single "master" model chooses which ones are most relevant to participate, and you only activate them.
In MoE systems the routing decision is made per-token, not per prompt or task. It’s one of ML’s many confusing naming conventions.
Even more confusingly, there are older pre-LLM MoE systems which ensemble and pool the predictions from multiple sub-components. For example in a random forest you could take the majority vote of the decision trees or the average of their numerical predictions.
After that, we developed neural net architectures for predicting a single thing like whether the user will click on your ad. An MMoE is in the same family.
And so now we are at massive MoE networks for LLMs which have similarities with MMoE in that the “decision” is about the very next token to predict.
Thanks for checking the individual rows. HumanEval+ is one small code slice, not the whole basis for the launch claim. The public evaluator currently contains 907 rows across seven benchmark families, and matched SWE-bench Verified and BigCodeBench runs are the next code evidence being added.
You also found a real UI bug: the inspector should show both stored answers and currently does not in some rows. We are fixing that.
On the row you reran: the page records a frozen matched run. It does not claim that Fable is incapable of solving that prompt on another run. We are adding repeated matched trials and making run count and variance explicit. Your rerun is exactly the kind of external check the row-level page is intended to make possible.
The broader result remains: Echo is competitive with Fable across the evaluated task mix at materially lower measured inference cost. We are filling in the harder agentic-code evidence now rather than asking anyone to infer it from HumanEval+.
I think approaches like this have potential. Only time will tell. This reminds me of the mixture of experts taken by deepseek r2 (I think it was r2, at least), but less specific models I guess.
I have often wondered how tools like GHCP choose the best model for the job when set to "auto".
such a scam, there is only one fable-like model, that somewhat behind, it cost half, not 3x. so from here you can stop reading.
I’m very confused on what this is, my initial thought was “oh nice, open source router.”
I go to the website…and it’s a sign up. I expected a repo. Otherwise how do I use it? As a SaaS? Yeah right.
Oh well I guess at least the benchmarks are good…I find the benchmarks and many are either not present or are not what the title claims.
My main question is how this has so many updoots from HN, probably the passerby not looking closer for sure.
I mean no offense and I really do wish you best on this, but it seems like what we used to call back in the day, vaporware.
This reminds me on OpenRouters report that combining multiple different models gave comparable performance to Fable 5. I think this approach has lots of potential. Maybe OpenAi was ahead of its time with GPT-5 (it being a router to different models rather than just being one new model)
You might be interested in a project that I'm working on, which is kind of like OpenRouter Fusion, but instead of waiting for all models to synthesize, we stream the best model immediately and background the rest. The background models then reconcile with an emoji reaction and optional reply. It gets similar results to Fusion and is a lot faster! There's a free version that leverages open weight models here: http://pellmell.ai.
M-o-MoE
I don’t find the recent spate of blog posts and systems delegating and combining LLMs to get better performance particularly interesting. Especially given that anyone who’s taken an ML 101 course has learned about ensemble methods.
While an LLM isn’t what you’d traditionally consider a weak learner, the theorems on learning systems clearly point to them being so in this context. The feigned surprise at combining them to yield better results seems disingenuous.
Even so, the work to predict which models are best suited for which task, how to delegate, and how to combine their outputs is interesting, especially if you’re placing a cost minimization objective on it. That said, this isn’t too far off from what many AI labs are already doing.
The entirety of "agents" and tool calls is a process of combining LLMs to get better results. Is it the same LLM in many cases? Yes. But it doesn't have to be.
It's the natural move that happened after people realized you couldn't throw away half a century of AI research.
Most of these focus on costs. But it is simply the case that the one-shot output did not scale for harder problems on workflows.
how does this differs from OpenRouter fusion?
how does this differs from Sakana Fugu?
Fugu includes closed source models in its pool
Fable-level, yea, but can it run gstack?
That’s basically the same idea IBM advertises with Bob?
Might want to rethink the name to avoid an Amazon issue.
If you copy Perplexity, they let you have the first few rounds of chat for free to get you going before asking to sign up.
Is this yet another Sakana Fugu / OpenRouter Fusion?
Looks like it but with open-weights only.