It started with a simple experiment. I took a group of models, including GLM-5.2, Kimi K2.7 and others, and ran them on the same evaluations. Then I measured what would happen if, for each problem, you somehow knew in advance which models would be useful and how their outputs should be combined.
That hypothetical system performed substantially better than any individual model in the pool. Of course, it is not something you can actually deploy because it relies on knowing which decisions were good after seeing the result. Echo is my attempt to recover some of that advantage without having that information in advance.
For each request, Echo decides how much computation to allocate, which models should participate, and how their work should be combined. Some prompts may only need a relatively small amount of inference, while others benefit from multiple models working on different parts of the problem.
One thing that surprised me while building it was how complementary the models are. A model that is clearly weaker overall can still be extremely useful on particular problems or as part of a combination.
On my first evaluation mix, Echo consistently performed better than the best individual model in its pool. It also reached roughly the same aggregate result as Fable, which I used as one of the stronger comparison systems, at around one third of the inference cost.
There are still some cases where Echo makes the wrong allocation or combination decision. I’m currently spending a lot of time understanding those failures, as well as testing whether the same approach holds up on coding and agentic tasks where measuring the quality of each decision becomes much harder.
I built a chat interface (echo.tracerml.ai) and an OpenAI-compatible API (https://echo.tracerml.ai/docs/api) so the system can be tested outside the evaluation setup.
Here is a short/high level video on how it works: https://www.youtube.com/watch?v=lJFJSvOdXhg
I wrote up the evaluation methodology, individual model results, costs and current limitations here: https://echo.tracerml.ai/eval
I would love for you to try it! Especially if you hit any weird failure cases or places where the allocation looks unintuitive.
I've literally taken one step on your website - the one your site design invited me to take - and immediately got tripped up. I'm not coming back.
Since you are fixing dark patterns, can you also put data privacy links somewhere other than on the gdpr banner?
Once clicked is impossible to see them, or at least I could not find them.
I dont know, it's something I can understand as someone also building AI stuff
This dark pattern is reminiscent of those online test sites in the 2000's where you spend 10 minutes filling out some quiz, then get prompted for an email address to see the results.
https://chat.mistral.ai/chat <- let me chat and actually responded without signing up.
MoonshotAI had this fake chat box dark pattern.
So I signed up with Mistral instead.
i am going to try to address a couple of topics that came up often:
- i'll keep publishing stronger evals, including more difficult coding and agentic benchmarks, to map out more precisely the differences with sota
- the public eval dashboard will keep expanding and be updated (very open to more benchmark suggestions as well!)
- some people found issues in the eval dashboard ui and the sign up flow, should be now all fixed in prod
some important precisions as well:
- NO credit card is required to try Echo
- each acount includes 10$ of free credits to try on both the API and the chat
on the approach itself: the idea i'm exploring is more broader than model routing, i'm looking at how to allocate inference efficiently across open-weight models, deciding not only which models to use, but also how much computation a request deserves and how intermediate work should be combined.
ensembling by itself is not new. since random forests and probably even before in statistics/classic ml we knew that bringing multiple models together can outperform individual ones. the interesting problem for Echo is how to model and leverage this without paying the full ensemble cost at each request.
while there are conceptual similarities with systems like Fusion or Fugu, the architecture and optimization objective are different.
thanks again for all the thoughtful feedback.
This isn't required either. Of course there's an xlcd for that: https://xkcd.com/936/
Besides,
--- start quote ---
Using complexity requirements (that is, where staff can only use passwords that are suitably complex) is a poor defence against guessing attacks. It places an extra burden on users, many of whom will use predictable patterns (such as replacing the letter ‘o’ with a zero) to meet the required 'complexity' criteria.
https://www.ncsc.gov.uk/collection/passwords/updating-your-a...
--- end quote ---
Oh, I'm well aware (with the concept and comic both). My passwords that I set for myself almost always are like that. It's just that I wasn't sure if OP was aware of it, due to what their site was asking.
Seriously though, the best password boxes are one that have an entropy meter and check that your password has never been in a breach.
https://blog.codinghorror.com/password-rules-are-bullshit/
You had my interest, now not anymore.
I am guessing this is not targeting those of us on the heavily subsidized $200/mo plans. Sure, these plans may be temporary, but none of us really know how temporary they are. Until then, 1/3rd of the published API pricing is not very appealing.
I have in mind 50k-100k ish for 3d studio max or was it softimage? (Well seems softimage https://www.awn.com/animationworld/siggraph-news-announcing-... ).
So... Basically we are back to this era.
The value was the multiple CDs of MSDN documentation and code samples which where very handy considering the slim pickings on the internet in 1995.
These weren't included with an IDE perpetual license retail box.
The other weird thing about the subs is that if the agents aren't grinding if feels like I'm losing money.
And it was damn cheap too! The main page cost was like 5 dollar.
Dude that topped Meta's tokenmaxxxing board before it was shut down used 265 billion tokens in a month. I kid you not.
I can not imagine some shelling out $200/month and then using that product lightly.
The people paying for the plan are not the same people using it.
Of 6 people I have data on the $200/plan only 2 regularly use more than $400 value.
> I'll bet the other way: the plan is not cost effective unless you are coding
The person I've personally seen use the most tokens isn't a coder. They do the "second brain" thing and wow it uses a lot of tokens.
They believe in the value, and TBH I've seen them do some pretty interesting and impressive things with it.
> the most junior developer, so green they almost need mowing, are going to throw the agents into a loop
I think this is also true.
But loops actually hit the cache a lot and most people who are calculating the value they are getting from a subscription aren't taking this into account.
SemiAnalysis published a snippet of their analysis, and they believe their tokens are an effective price of $0.99/million, rather than $20/million the naive pricing calculation would give you.
It's not a 98% margin loss if your users are unwilling to pay 50 times the cost that they were previously paying, and if they have other options like open source providers. The calculus isn't so simple because some portion of users would switch to API, and so it's about how many would continue using the service rather than leaving for a competitor.
I'm aware they need to recoup the enormous cost of training and data centers, but on a purely inference cost level I'm not convinced that the 200 dollar plans are unprofitable.
Especially considering not everyone is tokenmaxxing, and in most parts of the world people take leave and companies do not cut their subscriptions.
I suspect they are priced to have a lifetime average price/token amount that is roughly break-even, or maybe a slight loss leader.
> have seen Dario say in multiple interviews that they are profitable on inference, which maybe he was only meaning to refer to API usage, but that's not the impression I got.
I think he does mean API usage. Don't forget they can (and do) adjust the number of tokens you get on each plan at any time to adjust their margins on those.
That means he knows that is controllable, and it only the underlaying inference that defines the succes or otherwise of the company.
Exactly. I have the Claude $100/mo plan, and use it moderately for open source hobby stuff. I still haven't dipped my toes into the Fable pool, but I always use Opus 4.8 on xhigh, and I never hit my limits.
On the other hand, though, there have been times when I've looked at /usage for a long-running session (e.g., 7-10 days, after it's compacted a few times), and it showed I'd used ~$450 worth of tokens just for that session. So I'm clearly getting value for the money here when it comes to the subscription cost. But I still don't hit limits, so...
Are your thought patterns worth 9800 dollars a month?
What's the RoR on analyzing those thought patterns?
Anthropic emailed me today:
The half usage limit still applies.
For Premium seats it is.
https://support.claude.com/en/articles/15424964-claude-fable...
the supercycle is on device models, and one of those evolutions is models baked into chip die, and you just upgrade chipsets every few years instead
so it's the hyperscalers that will take the L in that environment
Else, you break the cache by doing a round robin of the same conversation across different models. Likely you'll end up paying more than what it would've cost with a cache aware system
Anyhow, this kinda reminds me of that quote about architecture: "We replaced our monolith with micro services so that every outage could be more like a murder mystery."
It currently exposes 907 stored rows across seven benchmark families, with prompts, outputs, grades, and cost records. More benchmarks are coming soon.
Echo does not disclose its per-request routing decision because that policy is the product. We can, however, publish some of the eligible open-weight model pool, version dates, aggregate allocation mix, and evaluation settings without exposing the request-level recipe.
New video is also being made.
My honest advice: that's going to pull away a decent amount of potential customers, even though I think your idea/concept is fantastic.
For example, if we were to consider it for Canva, observerability and full transparency is critical requirement; we can't accept not knowing which model serves a request. Both for legal/contract reasons, co-ordinated capacity planning with API providers, or even just evaluating our prompts and harnesses; and debugging/tracing results that went wrong. So that renders it out of consideration; and also suggests some kind of adversarial relationship where customers aren't trusted with critical information.
I definitely understand you need to keep business value, but I don't think hiding which model a request is routed to, is the right one, or at least if you want to expand to bigger potential customers / more advanced LLM deployments.
I have not had need for a router product thus far so excuse my ignorance if this is standard, but how could I possibly use and improve a product built on a router like this if I am not permitted to see which model served my request? If I got a bad answer back in my LLM-powered app, do I really have no way of knowing which model was responsible?
The benchmarks are here → https://echo.tracerml.ai/eval/
They are not good benchmarks but at least they exist.
In my project, I wasted a huge amount of time trying to improve GPQA Diamond results above ~93% range. I realized my mistake when Fable dropped and made no improvement on this benchmark vs. Opus.
I just wish this were solving an actual problem rather than being a fairly transparent attempt to say something approximating, "Hey VCs, OpenRouter just became a unicorn but I can basically vibe code it"
Calling it "Fable-level" feels intellectually lazy / dishonest, but then again, what do you expect when there's so much money on the table.
https://github.com/SakanaAI/fugu
[1] https://www.notdiamond.ai/
This space is so crowded it feels like I see a new "model router" pop up every few weeks.
"microservices turn function calls into distributed computing problems"
seem very confusing to grug
Eval tests while giving general indicators might not be similar for each use case.
You needed to search all of them to find something decent.
That's roughly analogous to today. Ignoring cost, you'd be way better off asking all the LLMs to solve a problem (like coding) where you can verify the answer.
So the question is, for things like that -> can a group of models perform better than frontier models, especially at a reasonable cost?
Fable is not a great value, so unless you're trying to find answers to Erdos questions, you can probably do better on cost.
You can probably typically ask 3 or 4 of the top Chinese models for an answer and get a response for the same Fable question... Given that Fable isn't that much better, it's not surprising you can do better for a large subset of problems.
Precisely.
The more vague and non committal and hand-wavey and subjective the field for AI to answer, the better the results (imo).
Because the correct response to that query is "I have no idea -- you will need to provide more information"
and LLM Agents suck at that.
Well, search engines are trash again. Perhaps it should come back
That's mostly because the ability to make money on the web made the web trash.
When you couldn't monetize your websites, everyone's websites were passion projects.
Now everyone is trying to figure out how they can make $8M a year off yet another recipe website.
High quality, fast & cheap (all 3 combined) - is a formula success.
It’s just way easier said than done.
we did the same: https://trustedrouter.com/blog/prometheus-2-new-draco-state-...
https://en.wikipedia.org/wiki/Ensemble_learning
Did OP invent an “intelligence router?”
Echo also starts users with free credit and does not require a credit card to try it. The current signup flow did not make that clear enough, so we are fixing that presentation too.
Thanks for calling both out.
The industry has largely focused on building larger models, but your results suggest that intelligently routing requests to the right combination of specialized models can deliver greater gains at a much lower cost.
It also reinforces the idea that weaker models are not necessarily obsolete. They may simply excel in different areas and become much more valuable when combined with others. I'm curious to see whether this still holds for coding and agentic tasks, where choosing the right models is likely much more challenging.
on agentic and coding what's make the problem even deeper is the granularity. how and when to use each model and at which layer of abstraction (session, goal, task, turn/tool calling). this is also something we are working on actively!
Fusion generates many replies then synthesizes. This adds a ton of latency and cost, so it's going to be better only for cases where you're willing to wait a lot and pay a lot more.
Routers (like this project) are a different thing, they can theoretically improve performance and cost at the same time without increasing latency much. I'm a bit skeptical though, since knowing which LLM is going to be better on a cost adjusted basis is hard (see https://artificialanalysis.ai/models/capabilities/coding?cos..., where the cost per task vs. performance is not what you expect, for example comparing Qwen 3.7 Max to GPT Sol.
A project I'm working on is aimed at improving performance without added latency but from a different angle. Instead of waiting for all replies for synthesis (like OpenRouter Fusion), it streams the "best" reply immediately (using a router to pick the best model) then synthesizes with emoji reactions and optional replies from the background models. It's free to use here with no login: http://pellmell.ai
in short, all of them are token diarrhea. Annoyingly logorrheic. they output so much useless stuff that it comes to the point that it makes me thing if that's not on purpose.
it was not because of price, but quality, I started testing other models and honestly, glm 5.2 is FAR SUPERIOR than fable 5 in every aspect. It comes to a surprise when glm 5.2 does not finish the task successfully. Kimi k2.7 does require a bit more of guidance but still better experience than opus 4.8. I have not yet the chance of trying k3.
openai latest model are.... ridiculously bad at software design and implementation.
(note i am only speaking to the domain of my work which has lots of data analysis, machine learning and software engineering.)
I can't get it to review win32 unsafe rust code.
Make it make sense.
I find that very off-turning!
And still Fable beats it hands down 8-0 in one of them, and is at worst even in some others.
Also it doesn't make logical sense: A router can save costs, yes, but not magically be "smarter" somehow.
That's like selling "free energy".
Literally promising frontier-equal results but at 1/3 price, i.e. cheaper than Kimi K3?
Doesn't offer any real benchmarks or explanation of how this magic trick is accomplished. No credible team or notable scientists behind it...
I have often wondered how tools like GHCP choose the best model for the job when set to "auto".
For example, compute X tokens with model A, then feed those into model B, etc. to get chain of thought through a diverse set of mdoels rather than chain of thought through a heterogeneous chain.
Humans seem to strongly believe echo chambers are bad. Are LLMs the same?
I'm curious though if these training methods are convergent or are models actually different; just like how in the stock market people think they're "diversified" but the truth is their exposure is likely much more risk correlated than one might think.
In certain situations, one right answer is better than a committee discussing the problem, but in others its sometimes nice to have some alternative methods of solving something. Fun project nonetheless.
My approach to using multiple models has been less about CoT but more about time to first token, and how you can use a small model to start interacting with the user while in parallel the more complex model is building a larger more complex thought. My work on this was primarily for voice backed interfaces before the voice models became quite a lot faster.
As I understand it, in an MoE model, you essentially have hundreds of smaller sub-models ("experts") that are good at different tasks, and for every generated token, a single "master" model chooses which ones are most relevant to participate, and you only activate them.
Even more confusingly, there are older pre-LLM MoE systems which ensemble and pool the predictions from multiple sub-components. For example in a random forest you could take the majority vote of the decision trees or the average of their numerical predictions.
After that, we developed neural net architectures for predicting a single thing like whether the user will click on your ad. An MMoE is in the same family.
And so now we are at massive MoE networks for LLMs which have similarities with MMoE in that the “decision” is about the very next token to predict.
Have there been experiments with doing it per task? Like, "oh this is python project, use this model" "oh this is about writing fantasy, use this"?
I think OpenAI already has (had?) a feature like this called “auto” mode for thinking.
So “1/3 the cost” really depends.
Don't want to derail what you're trying to do with Echo in case I'm wide of the mark, but yeah even in that case, if you hadn't considered that use case for it, I reckon there will, probably inside six months, be a substantial market for non-technical users who are sick of seeing ads in a service they already pay a subscription for, and who don't care what the underlying model is - or, indeed, don't even understand the concept of an "underlying model" because they interface with AI as a product.
You have already taken the HFG idea way further than I had even thought of yet, and I feel vindicated in seeing someone else do it. I wish you the very best!
The frontier providers aren't dumb. They charge what they charge because they know this. If you think Fable is too expensive then the type of problems you are solving don't demand that level of capability.
If you are working on something cutting edge, something truly novel, the cost of frontier AI is well worth its price.
With all that being said. No one is going to complain if we can get the same capability at a lower cost. And I mean true parity. Not trading space for time.
But we get ~$2500/mo worth of Fable credits for $200/mo on Anthropic pan? I'm still confused why people (who don't have to use API billing) are chasing open weight models based on cost.
The $200 odd plans are already out of reach of many, many people.
The attrition of customers if they were to get rid of these subscriptions plans would be untenable.
The $200 plans are priced so that the power-users use them and then advocate about how great the product is. If you're buying a $200 plan, you're not doing it because of the price point but rather because of the amount of work it is doing for you.
https://artificialanalysis.ai/agents/coding-agents#artificia...
Could you please stop posting unsubstantive comments and flamebait? You've unfortunately been doing it repeatedly. It's not what this site is for, and destroys what it is for.
If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.
This would be like trying to outlaw Linux or peer-to-peer file sharing. It's technically possible to write and pass a law, but it's basically impossible to enforce it.
While an LLM isn’t what you’d traditionally consider a weak learner, the theorems on learning systems clearly point to them being so in this context. The feigned surprise at combining them to yield better results seems disingenuous.
Even so, the work to predict which models are best suited for which task, how to delegate, and how to combine their outputs is interesting, especially if you’re placing a cost minimization objective on it. That said, this isn’t too far off from what many AI labs are already doing.
It's the natural move that happened after people realized you couldn't throw away half a century of AI research.
Most of these focus on costs. But it is simply the case that the one-shot output did not scale for harder problems on workflows.
I go to the website…and it’s a sign up. I expected a repo. Otherwise how do I use it? As a SaaS? Yeah right.
Oh well I guess at least the benchmarks are good…I find the benchmarks and many are either not present or are not what the title claims.
My main question is how this has so many updoots from HN, probably the passerby not looking closer for sure.
I mean no offense and I really do wish you best on this, but it seems like what we used to call back in the day, vaporware.
Plus Fable is way less annoying to talk to than Opus 4.8. Opus 4.8's writing style is absolutely insufferable. Fable has some of the same quirks but it's way less bad.
https://www.ycombinator.com/companies?query=tracerml
I don't see it?
But in the present case, they're just a startup in the current batch.