This project started about three years ago as a small piece of a proxy project (ramaproxy.org): on every request we pick a random upstream proxy that matches a handful of flags and a country or city. The data barely changes, the queries happen constantly. Scanning a Vec got slow, and SQLite felt like a lot of database for "give me a random row with these flags".
VennDB is a Rust library: you add a derive macro to a struct, and it generates a database type and a query builder for it. Bools become bitmasks, filter values become sets of rows, and a query is the intersection of those sets. It can pick a random match without finding all of them first.
It is mostly a bitmap index, generated from your struct so you don't have to write and maintain it yourself. It is also narrow on purpose: append-only and in-memory, with no updates or deletes. As such it is not really a database in the traditional sense of the word, but as I was not sure how to name it differently given it does fulfill a niche subset of usecases that I would otherwise use an (in-memory) database for it seemed fitting.
We just released 0.7, which rewrote the engine. In the README compares it with a hand-written roaring bitmap index, SQLite and a plain Vec. roaring gets close on some queries. We're not database experts, so if the benchmarks look unfair or there's prior art we missed, I'd like to hear it.
Hi HN, I'm the author.
This project started about three years ago as a small piece of a proxy project (ramaproxy.org): on every request we pick a random upstream proxy that matches a handful of flags and a country or city. The data barely changes, the queries happen constantly. Scanning a Vec got slow, and SQLite felt like a lot of database for "give me a random row with these flags".
VennDB is a Rust library: you add a derive macro to a struct, and it generates a database type and a query builder for it. Bools become bitmasks, filter values become sets of rows, and a query is the intersection of those sets. It can pick a random match without finding all of them first.
It is mostly a bitmap index, generated from your struct so you don't have to write and maintain it yourself. It is also narrow on purpose: append-only and in-memory, with no updates or deletes. As such it is not really a database in the traditional sense of the word, but as I was not sure how to name it differently given it does fulfill a niche subset of usecases that I would otherwise use an (in-memory) database for it seemed fitting.
We just released 0.7, which rewrote the engine. In the README compares it with a hand-written roaring bitmap index, SQLite and a plain Vec. roaring gets close on some queries. We're not database experts, so if the benchmarks look unfair or there's prior art we missed, I'd like to hear it.