I've been working on a few storage engine designs recently implementing an LSM tree and would like to share K4.
K4 is a storage engine written completely in GO with no dependencies. The write speed surpasses RocksDB (7.8.3) with similar configuration.
I will write way more comprehensive benchmarks down the line.
The current features of K4
------------------------------
- High speed writes and reads
- Durability
- Variable length binary keys and values. Keys and their values can be any length
- Write-Ahead Logging (WAL). System writes PUT and DELETE operations to a log file before applying them to the LSM tree.
- Atomic transactions. Multiple PUT and DELETE operations can be grouped together and applied atomically to the LSM tree.
- Paired compaction. SSTables are paired up during compaction and merged into a single SSTable(s). This reduces the number of SSTables and minimizes disk I/O for read operations.
- Memtable implemented as a skip list.
- In-memory and disk-based storage
- Configurable memtable flush threshold
- Configurable compaction interval (in seconds)
- Configurable logging
- Configurable skip list
- Bloom filter for faster lookups. SSTable initial pages contain a bloom filter. The system uses the bloom filter to determine if a key is in the SSTable before scanning the SSTable.
- Recovery from WAL
- Granular page locking
- Thread-safe
- TTL (time to live) support
- Optional compression support (Simple lightweight and optimized Lempel-Ziv 1977 inspired compression algorithm
- Range and equi functionality (Get, NGet, Range, NRange, GreaterThan, GreaterThanEq, LessThan, LessThanEq)
- No dependencies
I am in the process of writing a C binding for K4 which will enable me to write FFI's for multiple other languages like Python, Node.JS, Ruby, etc.
I hope you get a chance to check it out and do let me know your thoughts!
You will inevitably face a lot of the typical questions like 'Why not just use RocksDB', 'Why is this necessary' etc. All I'd say is, don't get disheartened if/when that happens.
Maintaining an open source project is not easy and is often thankless, plus it takes a huge effort to get something of this complexity done. Thanks you for the effort!
Hey arunabha, I appreciate the comment! This isn't my first rodeo as they say :)
I love, love, love programming and have for over 15 years, nothing will change that. I enjoy sharing my ideas, and am a big open-source enthusiast. It's a good thing trying to be innovate. :)
Hello everyone! I hope you are all well.
I've been working on a few storage engine designs recently implementing an LSM tree and would like to share K4.
K4 is a storage engine written completely in GO with no dependencies. The write speed surpasses RocksDB (7.8.3) with similar configuration.
I will write way more comprehensive benchmarks down the line.
The current features of K4
------------------------------
- High speed writes and reads
- Durability
- Variable length binary keys and values. Keys and their values can be any length
- Write-Ahead Logging (WAL). System writes PUT and DELETE operations to a log file before applying them to the LSM tree.
- Atomic transactions. Multiple PUT and DELETE operations can be grouped together and applied atomically to the LSM tree.
- Paired compaction. SSTables are paired up during compaction and merged into a single SSTable(s). This reduces the number of SSTables and minimizes disk I/O for read operations.
- Memtable implemented as a skip list.
- In-memory and disk-based storage
- Configurable memtable flush threshold
- Configurable compaction interval (in seconds)
- Configurable logging
- Configurable skip list
- Bloom filter for faster lookups. SSTable initial pages contain a bloom filter. The system uses the bloom filter to determine if a key is in the SSTable before scanning the SSTable.
- Recovery from WAL
- Granular page locking
- Thread-safe
- TTL (time to live) support
- Optional compression support (Simple lightweight and optimized Lempel-Ziv 1977 inspired compression algorithm
- Range and equi functionality (Get, NGet, Range, NRange, GreaterThan, GreaterThanEq, LessThan, LessThanEq)
- No dependencies
I am in the process of writing a C binding for K4 which will enable me to write FFI's for multiple other languages like Python, Node.JS, Ruby, etc.
I hope you get a chance to check it out and do let me know your thoughts!
Thank you kindly.
Congratulations on getting this out!
You will inevitably face a lot of the typical questions like 'Why not just use RocksDB', 'Why is this necessary' etc. All I'd say is, don't get disheartened if/when that happens.
Maintaining an open source project is not easy and is often thankless, plus it takes a huge effort to get something of this complexity done. Thanks you for the effort!
Hey arunabha, I appreciate the comment! This isn't my first rodeo as they say :)
I love, love, love programming and have for over 15 years, nothing will change that. I enjoy sharing my ideas, and am a big open-source enthusiast. It's a good thing trying to be innovate. :)