3 comments

  • alexpadula 5 hours ago

    Hello everyone! I hope you are all well.

    I've been working on a few storage engine designs recently implementing an LSM tree and would like to share K4.

    K4 is a storage engine written completely in GO with no dependencies. The write speed surpasses RocksDB (7.8.3) with similar configuration.

    I will write way more comprehensive benchmarks down the line.

    The current features of K4

    ------------------------------

    - High speed writes and reads

    - Durability

    - Variable length binary keys and values. Keys and their values can be any length

    - Write-Ahead Logging (WAL). System writes PUT and DELETE operations to a log file before applying them to the LSM tree.

    - Atomic transactions. Multiple PUT and DELETE operations can be grouped together and applied atomically to the LSM tree.

    - Paired compaction. SSTables are paired up during compaction and merged into a single SSTable(s). This reduces the number of SSTables and minimizes disk I/O for read operations.

    - Memtable implemented as a skip list.

    - In-memory and disk-based storage

    - Configurable memtable flush threshold

    - Configurable compaction interval (in seconds)

    - Configurable logging

    - Configurable skip list

    - Bloom filter for faster lookups. SSTable initial pages contain a bloom filter. The system uses the bloom filter to determine if a key is in the SSTable before scanning the SSTable.

    - Recovery from WAL

    - Granular page locking

    - Thread-safe

    - TTL (time to live) support

    - Optional compression support (Simple lightweight and optimized Lempel-Ziv 1977 inspired compression algorithm

    - Range and equi functionality (Get, NGet, Range, NRange, GreaterThan, GreaterThanEq, LessThan, LessThanEq)

    - No dependencies

    I am in the process of writing a C binding for K4 which will enable me to write FFI's for multiple other languages like Python, Node.JS, Ruby, etc.

    I hope you get a chance to check it out and do let me know your thoughts!

    Thank you kindly.

    • arunabha 21 minutes ago

      Congratulations on getting this out!

      You will inevitably face a lot of the typical questions like 'Why not just use RocksDB', 'Why is this necessary' etc. All I'd say is, don't get disheartened if/when that happens.

      Maintaining an open source project is not easy and is often thankless, plus it takes a huge effort to get something of this complexity done. Thanks you for the effort!

      • alexpadula 17 minutes ago

        Hey arunabha, I appreciate the comment! This isn't my first rodeo as they say :)

        I love, love, love programming and have for over 15 years, nothing will change that. I enjoy sharing my ideas, and am a big open-source enthusiast. It's a good thing trying to be innovate. :)