shard fit is a nice touch. I just ran into this issue the other day, model was too big for gpu, started spilling inference over to cpu and responses were taking foreverrr. Out of curiosity, what happens if a worker drops mid-generation? Clean fail or timeout?
shard fit is a nice touch. I just ran into this issue the other day, model was too big for gpu, started spilling inference over to cpu and responses were taking foreverrr. Out of curiosity, what happens if a worker drops mid-generation? Clean fail or timeout?
Super cool idea
Thanks, I’m continuing to develop the project, check out the Git, you’ll like it.
[flagged]