Do they cache results so users can receive already generated answers?
If the resellers cache results, how it is done? I am aware it's possible with a KV cache, but those resellers do not have access to Claude's KV memory?
For example a user query is transformed to filter the request to and the answer from Claude with a local open source model.
Use case: A user asks to find a bug in code, they provide a large piece of code (hence a large amount of tokens), it is processed by the retailer with an open source LLM which is asked to extract suspicious snippets, and Claude is asked to provide an exact answer but as short as possible, then the local open source model is asked to incorporate the modifications in the original code.
The number of tokens used by Claude is much lower than if the user has asked directly to Claude. The local open source model provides the difference.
Nope, it’s just simple arbitrage. All of the subscription tiers offer 5-10x better prices than metered api fees. They use subscription to hook people and get them desperate enough to pay for ad-hoc generation outside of their subscription to make a profit. The economics of Claude do not work unless a significant % of subscribers convert to standard metered API use. They are trying any trick under the sun to make this happen.
Do they cache results so users can receive already generated answers?
If the resellers cache results, how it is done? I am aware it's possible with a KV cache, but those resellers do not have access to Claude's KV memory?
If they do not cache results, how does this work?
Could it be use of a tainted Claude?
For example a user query is transformed to filter the request to and the answer from Claude with a local open source model.
Use case: A user asks to find a bug in code, they provide a large piece of code (hence a large amount of tokens), it is processed by the retailer with an open source LLM which is asked to extract suspicious snippets, and Claude is asked to provide an exact answer but as short as possible, then the local open source model is asked to incorporate the modifications in the original code.
The number of tokens used by Claude is much lower than if the user has asked directly to Claude. The local open source model provides the difference.
Check previous discussion: https://news.ycombinator.com/item?id=49058993
My personal guess is some of those "providers" use inputs/outputs for training "open"weight models.
Nope, it’s just simple arbitrage. All of the subscription tiers offer 5-10x better prices than metered api fees. They use subscription to hook people and get them desperate enough to pay for ad-hoc generation outside of their subscription to make a profit. The economics of Claude do not work unless a significant % of subscribers convert to standard metered API use. They are trying any trick under the sun to make this happen.