That's kinda neat. Do you do any decoding tricks to handle varying processing capabilities across device models and ipads? Like shortening ctx length or maybe looking into the LFM models that have MoEs with like 1b active?
Right now I am just restricting it to devices with 8GB RAM. I plan to start testing older phones soon to see if I can get it running well on them. MoE with low active params is interesting. I wonder if there is smaller models based on specific pairs (En/Es, ect.)
That's kinda neat. Do you do any decoding tricks to handle varying processing capabilities across device models and ipads? Like shortening ctx length or maybe looking into the LFM models that have MoEs with like 1b active?
Right now I am just restricting it to devices with 8GB RAM. I plan to start testing older phones soon to see if I can get it running well on them. MoE with low active params is interesting. I wonder if there is smaller models based on specific pairs (En/Es, ect.)