I’ve been using local models on a Framework desktop. I get > 50 tokens/s on Qwen 3.6 35B A3B, and I find that speed ok for coding (though the model not quite). If machine could do a SOTA model with say 0.5-2T parameters like GLM 5.2 at 100 tok/s, it would be very useable.
In 2 years presumably the SOTA models will be much bigger, but I won’t be surprised if there’s a fable level sub 50B parameter model by that point either*, in which case 1000 tok/s may be reasonable.
* on artificial analysis intelligence index, GPT 4o is 12 and Qwen 35B A3B is 32.
Are we going to get 1000 tps local?
I’ve been using local models on a Framework desktop. I get > 50 tokens/s on Qwen 3.6 35B A3B, and I find that speed ok for coding (though the model not quite). If machine could do a SOTA model with say 0.5-2T parameters like GLM 5.2 at 100 tok/s, it would be very useable.
In 2 years presumably the SOTA models will be much bigger, but I won’t be surprised if there’s a fable level sub 50B parameter model by that point either*, in which case 1000 tok/s may be reasonable.
* on artificial analysis intelligence index, GPT 4o is 12 and Qwen 35B A3B is 32.