Cutting LLM Batch Inference Time in Half: Dynamic Prefix Bucketing at Scale

1 comments

• sammysidhu 8 hours ago

  Part of the Daft team here! Happy to answer any questions