What an incredibly useful feature. Besides the obvious developer experience benefits, it’s huge for network-bound use cases: really heavily optimized uses of RabbitMQ (or less-optimized uses with really big message payloads) end up bottlenecked or paying lots of money for broker network capacity, since a message’s bytes must cross the wire
2 or more times (publish, consume, maybe replication) for it to be processed. Moving filtering logic to the consumer side helps a lot with that—but workloads should still use separate queues/topics/streams instead whenever they can, of course (I’m sure there will be some one-topic-for-everything abuses enabled by the combination of poor architectural foresight + SQL filtering, but such is life).
I am confused, though: why does the bloom filter … er, filter still need to be manually specified by the consumer (filterValues in the example Java)?
As far as the broker filtering query evaluation logic is concerned, bloom-filter enabled fields are just indexes; why can’t the SQL-filter query planner automatically make use of them?
I’m probably missing something, but it seems like a very light query plan optimization pass would not be hard to implement here; there’s only one kind of index, and it can only be used with equality comparisons, so it doesn’t seem like the the implementation complexity would be too bad versus needing a fully general SQL optimizing plannner.
What an incredibly useful feature. Besides the obvious developer experience benefits, it’s huge for network-bound use cases: really heavily optimized uses of RabbitMQ (or less-optimized uses with really big message payloads) end up bottlenecked or paying lots of money for broker network capacity, since a message’s bytes must cross the wire 2 or more times (publish, consume, maybe replication) for it to be processed. Moving filtering logic to the consumer side helps a lot with that—but workloads should still use separate queues/topics/streams instead whenever they can, of course (I’m sure there will be some one-topic-for-everything abuses enabled by the combination of poor architectural foresight + SQL filtering, but such is life).
I am confused, though: why does the bloom filter … er, filter still need to be manually specified by the consumer (filterValues in the example Java)?
As far as the broker filtering query evaluation logic is concerned, bloom-filter enabled fields are just indexes; why can’t the SQL-filter query planner automatically make use of them?
I’m probably missing something, but it seems like a very light query plan optimization pass would not be hard to implement here; there’s only one kind of index, and it can only be used with equality comparisons, so it doesn’t seem like the the implementation complexity would be too bad versus needing a fully general SQL optimizing plannner.
I guess there's some effect on the broker side wrt resources or efficiency, but I couldn't immediately see anything about this.