Benchmarks for document tasks for local AI models

(bendyline.com)

1 points | by mikeamm 3 hours ago ago

1 comments

  • mikeamm 3 hours ago

    I created some basic benchmarks of document tasks and surveyed lower-end AI models like Qwen and Gemma to see how they compare to cloud models like Sonnet and Sol. In sum, the better local models runnable on normal h/w, like Gemma 31B or Qwen 3.8 27b, still scored about 7 points behind Claude Sonnet 5.5. Results also show the spectrum of quality between models that can run on a moderate GPU-less PC vs. a higher end GPU PC or a higher end Mac.