23 points | by ankitvgupta an hour ago ago
4 comments
Does not say what type of model or size they were serving, but one of the traces mentions Maple, which is a 20b model. Still, getting ~70+ tok/s is quite good.
Fpga is software-driven, model can verify its output easily.
Zero surprise it did it
it only cost them $10,000 in Claude API usage. :)
You get that much usage in a monthly $200 subscription.
Does not say what type of model or size they were serving, but one of the traces mentions Maple, which is a 20b model. Still, getting ~70+ tok/s is quite good.
Fpga is software-driven, model can verify its output easily.
Zero surprise it did it
it only cost them $10,000 in Claude API usage. :)
You get that much usage in a monthly $200 subscription.