4 comments

  • rraithel27 5 hours ago

    shard fit is a nice touch. I just ran into this issue the other day, model was too big for gpu, started spilling inference over to cpu and responses were taking foreverrr. Out of curiosity, what happens if a worker drops mid-generation? Clean fail or timeout?

  • tannerjames711 6 hours ago

    Super cool idea

    • avelone 6 hours ago

      Thanks, I’m continuing to develop the project, check out the Git, you’ll like it.

  • avelone 7 hours ago

    [flagged]