It actually is one of the more perfect features of the project. I was considering doing something like that myself in the early days of the project. It was around that time someone in Python land created a graphql-like mad libs language for this kind of thing. But the idea of doing it for llama.cpp using BNF syntax didn't occur to me at the time, until someone else came along and wrote it for me. Once I saw that, it became clear and obvious that this is the way. It's a best of all possible worlds kind of design. You could build an empire on that tool alone. For many months the grammar feature was the #1 feature advantage that llama.cpp had over OpenAI, which had nothing comparable. That's actually kind of a rare thing in open source since we're normally not the ones advancing the technological frontier. Similar to GIMP's color to alpha feature, which is the one thing it can do that Photoshop can't. If commercial opportunities exist for grammar, they will probably lie in democratizing and evangelizing the technology rather than improving upon it.
The llama.cpp --grammar flag just raised $12m from European investors.
This is actually pretty funny.
You're right, llamacpp has completely perfected it. No need to work on it anymore. We'll let them know.
It actually is one of the more perfect features of the project. I was considering doing something like that myself in the early days of the project. It was around that time someone in Python land created a graphql-like mad libs language for this kind of thing. But the idea of doing it for llama.cpp using BNF syntax didn't occur to me at the time, until someone else came along and wrote it for me. Once I saw that, it became clear and obvious that this is the way. It's a best of all possible worlds kind of design. You could build an empire on that tool alone. For many months the grammar feature was the #1 feature advantage that llama.cpp had over OpenAI, which had nothing comparable. That's actually kind of a rare thing in open source since we're normally not the ones advancing the technological frontier. Similar to GIMP's color to alpha feature, which is the one thing it can do that Photoshop can't. If commercial opportunities exist for grammar, they will probably lie in democratizing and evangelizing the technology rather than improving upon it.
You mean this https://lmsys.org/blog/2024-02-05-compressed-fsm/
It has a clever way to decode multiple valid tokens at once, rather than just one token at a time.
Corresponding project link: https://github.com/sgl-project/sglang
Crazy...