Kind of what I've been working on to build tenancy on top of SQLite CDC to make it a simple repayable SQLite for Marmot (https://github.com/maxpert/marmot). I personally think we have a synergy here, would drop by your discord.
Neat! Having literally everything backed by object storage is The Dream, so this makes a lot of sense. So to compare this to the options that are available (that aren't Kafka or Redis streams) I can imagine you could take these items that you're writing to a stream, batch them and write them into some sort of S3-backed data lake. Something like Delta Lake. And then query them using I don't know DuckDB or whatever your OLAP SQL thing is. Or you could what develop your own S3 schema that that's just saving these items to batched objects as they come in. So then part what S2 is saving you from is having to write your own acknowledgement system/protocol for batching these items, and the corresponding read ("consume") queries? Cool!
Yes, that is a reasonable way to think about it! And as s2-lite is designed as a single-node system, there is a natural source of truth on what the latest records are for consuming in real-time.
This is fair question. A stream here == a log. Every write with S2 implementations is durable before it is acknowledged, and it can be consumed in real-time or replayed from any position by multiple readers. The stream is at the granularity of discrete records, rather than a byte stream (although you can certainly layer either over the other).
ED: no k8s required for s2-lite, it is just a singe binary. It was an architectural note about our cloud service.
Your documentation needs improvement. It proudly mentions the alphabet soup of technologies you use, but it leaves me completely baffled about what s2 does, what problem s2 is trying to solve, or who the intended audience of s2 is.
So you frame the data into records, save the frame somehow (maybe with fsync if you're doing it locally, or maybe you outsource it to S3 or S3-compatible storage?), then ack and start sending it to clients. Therefore every frame that's acked or sent to clients has already been saved.
Personally I'd add an application level hash to protect the integrity of the records but that's just me.
At first glance I wondered if a hash chain or Merkle tree might be useful but I think it's overkill. What exactly is the trust model? I get the sense this is a traditional client-server protocol (i.e., not p2p). Does it stream the streams over HTTP / HTTPS, or some custom protocol? Are s2 clients expected to be end-user web browsers, other instances of s2 or something else?
> it leaves me completely baffled about what s2 does, what problem s2 is trying to solve, or who the intended audience of s2 is
Regarding S2 generally (not just s2-lite), the intent behind it is to turn the core data structure from streaming platforms (like Kafka) into a serverless primitive -- kinda similar to what object storage did for file storage.
So if you are already in the world of working with streaming platforms, S2 gives you a simpler API, bottomless storage (S2 itself uses object storage for durability), and no limits on the quantity of streams you can create and work with. Streams also all have URIs and are directly accessible over REST with granular access controls.
This enables new types of patterns for working with streams, other than just the traditional ones where people typically reach for streaming platforms (like CDC, ETL pipelines, etc). An agent can have its own stream to serialize state onto, for instance; you can use a stream as a durable transport layer -- e.g., you want to reliably provide a flow of data (tokens from a model, financial ticker data, etc) to a user and allow them to resume from exactly where they left off if they are disconnected, for instance; you could use streams as a durable ingest buffer, for collecting data that will eventually reside in an OLAP like Clickhouse.
> Personally I'd add an application level hash to protect the integrity of the records but that's just me.
The durability is for being able to replay the stream, a hash will not let you reconstruct the original message(s).
If you just need ephemeral comms, making it persistent is indeed overkill. But reliability challenges often come up with seemingly ephemeral comms too – think streaming responses from an LLM. The last mile can be pretty flaky e.g. iOS will cancel connections when users background an app. Using a durable stream for persisting the tokens means a client can ask to resume from where it left off / from the beginning of the stream, and the data would be available without having to re-inference.
Shoutout to CodesInChaos for suggesting that instead of a mere emulator, should have an actually durable open source implementation – that is what we ended up building with s2-lite! https://news.ycombinator.com/item?id=42487592
And it has the durability of object storage rather than just local. SlateDB actually lets you also use local FS, will experiment with plumbing up the full range of options - right now it's just in-memory or S3-compatible bucket.
> So I'd try so share as much of the frontend code (e.g. the GRPC and REST handlers) as possible between these.
Right on, this is indeed the case. The OpenAPI spec is also now generated off the REST handlers from s2-lite. We are getting rid of gRPC, s2-lite only supports the REST API (+ gRPC-like session protocol over HTTP/2: https://s2.dev/docs/api/records/overview#s2s-spec)
We wanted S2 to be one API. Started out with gRPC, added REST - then realized REST is what is absolutely essential and what most folks care about. gRPC did give us bi-directional streaming for append/read sessions, so we added that as an optional enhancement to the corresponding POST/GET data plane endpoints (the S2S "S2-Session" spec I linked to above). A nice side win is that the stream resource is known from the requested URL rather than having to wait for the first gRPC message.
gRPC ecosystem is also not very uniform despite its popularity, comes with bloat, is a bit of a mess in Python. I'm hoping QUIC enables a viable gRPC alternative to emerge.
Can this be used as an embedded lib instead of a separate binary as an API?
And am I understanding correctly that if I pointed 2 running instances of s2-lite at the same place in s3 there would be problems since slatedb is single writer?
Happy to accept contributions that make this more ergonomic.
> And am I understanding correctly that if I pointed 2 running instances of s2-lite at the same place in s3 there would be problems since slatedb is single writer?
SL8 will fence the older writer, thanks to S3 conditional writes. I think there would be potential for stale reads until the fencing happens...
Love this. Elegant and powerful. Stateful streams are surprisingly difficult to DIY and as everything becomes a stream of tokens this is super useful tool to have in the toolbox.
Kind of what I've been working on to build tenancy on top of SQLite CDC to make it a simple repayable SQLite for Marmot (https://github.com/maxpert/marmot). I personally think we have a synergy here, would drop by your discord.
Very cool! Looking forward to catching up. Also checkout https://s2.dev/blog/kv-store
Neat! Having literally everything backed by object storage is The Dream, so this makes a lot of sense. So to compare this to the options that are available (that aren't Kafka or Redis streams) I can imagine you could take these items that you're writing to a stream, batch them and write them into some sort of S3-backed data lake. Something like Delta Lake. And then query them using I don't know DuckDB or whatever your OLAP SQL thing is. Or you could what develop your own S3 schema that that's just saving these items to batched objects as they come in. So then part what S2 is saving you from is having to write your own acknowledgement system/protocol for batching these items, and the corresponding read ("consume") queries? Cool!
Yes, that is a reasonable way to think about it! And as s2-lite is designed as a single-node system, there is a natural source of truth on what the latest records are for consuming in real-time.
When someone says "stream data over the Internet," my automatic reaction is "open a TCP connection."
Adding a database, multiple components, and Kubernetes to the equation seems like massively overengineering.
What value does S2 provide that simple TCP sockets do not?
Is this for like "making your own Twitch" or something, where streams have to scale to thousands-to-millions of consumers?
This is fair question. A stream here == a log. Every write with S2 implementations is durable before it is acknowledged, and it can be consumed in real-time or replayed from any position by multiple readers. The stream is at the granularity of discrete records, rather than a byte stream (although you can certainly layer either over the other).
ED: no k8s required for s2-lite, it is just a singe binary. It was an architectural note about our cloud service.
Your documentation needs improvement. It proudly mentions the alphabet soup of technologies you use, but it leaves me completely baffled about what s2 does, what problem s2 is trying to solve, or who the intended audience of s2 is.
So you frame the data into records, save the frame somehow (maybe with fsync if you're doing it locally, or maybe you outsource it to S3 or S3-compatible storage?), then ack and start sending it to clients. Therefore every frame that's acked or sent to clients has already been saved.
Personally I'd add an application level hash to protect the integrity of the records but that's just me.
At first glance I wondered if a hash chain or Merkle tree might be useful but I think it's overkill. What exactly is the trust model? I get the sense this is a traditional client-server protocol (i.e., not p2p). Does it stream the streams over HTTP / HTTPS, or some custom protocol? Are s2 clients expected to be end-user web browsers, other instances of s2 or something else?
> it leaves me completely baffled about what s2 does, what problem s2 is trying to solve, or who the intended audience of s2 is
Regarding S2 generally (not just s2-lite), the intent behind it is to turn the core data structure from streaming platforms (like Kafka) into a serverless primitive -- kinda similar to what object storage did for file storage.
So if you are already in the world of working with streaming platforms, S2 gives you a simpler API, bottomless storage (S2 itself uses object storage for durability), and no limits on the quantity of streams you can create and work with. Streams also all have URIs and are directly accessible over REST with granular access controls.
This enables new types of patterns for working with streams, other than just the traditional ones where people typically reach for streaming platforms (like CDC, ETL pipelines, etc). An agent can have its own stream to serialize state onto, for instance; you can use a stream as a durable transport layer -- e.g., you want to reliably provide a flow of data (tokens from a model, financial ticker data, etc) to a user and allow them to resume from exactly where they left off if they are disconnected, for instance; you could use streams as a durable ingest buffer, for collecting data that will eventually reside in an OLAP like Clickhouse.
> Personally I'd add an application level hash to protect the integrity of the records but that's just me.
The durability is for being able to replay the stream, a hash will not let you reconstruct the original message(s).
If you just need ephemeral comms, making it persistent is indeed overkill. But reliability challenges often come up with seemingly ephemeral comms too – think streaming responses from an LLM. The last mile can be pretty flaky e.g. iOS will cancel connections when users background an app. Using a durable stream for persisting the tokens means a client can ask to resume from where it left off / from the beginning of the stream, and the data would be available without having to re-inference.
> Is this for like "making your own Twitch" or something, where streams have to scale to thousands-to-millions of consumers?
Yes, this can be a good building block for broadcasting data streams.
s2-lite is single node, so to scale to that level, you'd need to add some CDN-ing on top.
s2.dev is the elastic cloud service, and it supports high fanout reads using Cachey (https://www.reddit.com/r/databasedevelopment/comments/1nh1go...)
Shoutout to CodesInChaos for suggesting that instead of a mere emulator, should have an actually durable open source implementation – that is what we ended up building with s2-lite! https://news.ycombinator.com/item?id=42487592
And it has the durability of object storage rather than just local. SlateDB actually lets you also use local FS, will experiment with plumbing up the full range of options - right now it's just in-memory or S3-compatible bucket.
> So I'd try so share as much of the frontend code (e.g. the GRPC and REST handlers) as possible between these.
Right on, this is indeed the case. The OpenAPI spec is also now generated off the REST handlers from s2-lite. We are getting rid of gRPC, s2-lite only supports the REST API (+ gRPC-like session protocol over HTTP/2: https://s2.dev/docs/api/records/overview#s2s-spec)
> We are getting rid of gRPC
I'm curious why and what challenges you had with gRPC. s2-lite looks cool!
We wanted S2 to be one API. Started out with gRPC, added REST - then realized REST is what is absolutely essential and what most folks care about. gRPC did give us bi-directional streaming for append/read sessions, so we added that as an optional enhancement to the corresponding POST/GET data plane endpoints (the S2S "S2-Session" spec I linked to above). A nice side win is that the stream resource is known from the requested URL rather than having to wait for the first gRPC message.
gRPC ecosystem is also not very uniform despite its popularity, comes with bloat, is a bit of a mess in Python. I'm hoping QUIC enables a viable gRPC alternative to emerge.
Can this be used as an embedded lib instead of a separate binary as an API?
And am I understanding correctly that if I pointed 2 running instances of s2-lite at the same place in s3 there would be problems since slatedb is single writer?
> Can this be used as an embedded lib instead of a separate binary as an API?
Did not architect explicitly for that, but should be viable. You could use the `Backend` directly, is what the REST handlers call https://docs.rs/s2-lite/latest/s2_lite/backend/struct.Backen...
Happy to accept contributions that make this more ergonomic.
> And am I understanding correctly that if I pointed 2 running instances of s2-lite at the same place in s3 there would be problems since slatedb is single writer?
SL8 will fence the older writer, thanks to S3 conditional writes. I think there would be potential for stale reads until the fencing happens...
ED: Fresh discussion in https://discord.com/channels/1232385660460204122/12323856609...
The stale read potential can be mitigated, https://github.com/s2-streamstore/s2/issues/91
Love this. Elegant and powerful. Stateful streams are surprisingly difficult to DIY and as everything becomes a stream of tokens this is super useful tool to have in the toolbox.
Would be useful to have SlateDB WAL go to Valkey or somewhere else to reduce s3 put costs and latency.
It seems SL8 supports writing the WAL to local disk already https://github.com/slatedb/slatedb/issues/162
Will look into how to enable that option from s2-lite