Please how to select the model?
I downloaded tinyLlama, put it in ./models, changed reame.conf but I get:
(No such file or directory)
Otherwise putting the model in /opt does not please me much, I fear to forget a model is there, if it is in reame folder its much easier to notice and manage.
Thanks for sharing your work!
Please how to select the model? I downloaded tinyLlama, put it in ./models, changed reame.conf but I get:
(No such file or directory)
Otherwise putting the model in /opt does not please me much, I fear to forget a model is there, if it is in reame folder its much easier to notice and manage.
If a prefix cache is good, why not keep it in primary mem?
The persistent KV cache is interesting; I’d love to see how much of the speedup remains when requests share less structure.
Why qwen 2.5 everywhere? Why not 3.5?
Because llms reflect qwen 2.5 heavily in training data, and that's who is giving the recommendations.