A naukri.com initiative
Dev
2w
120
Image Credit: Dev
Introducing Gridstore: Qdrant's Custom Key-Value Store
- Qdrant built its custom storage engine called Gridstore, to address issues in RocksDB, which required compaction, handles generic keys, and took time to configure due to different options. Qdrant wanted better control over when and which data was written and flushed. It did not demand generic keys, and already has optimizers for online compaction. Full control over how data is stored is achieved by building its own storage expediting performance irrespective of memory status or any other issue.
- Qdrant's custom storage engine, Gridstore is composed of three layers: The Data Layer, The Mask Layer, and The Gaps Layer. The Data Layer stores and retrieves variable-sized data quickly based on specific keys, and its array of pointers helps it to retrieve the data significantly faster than generic methods. While storing the data, Gridstore categorizes each data in fixed-sized blocks, and these blocks are further grouped into larger page files.
- The Mask Layer helps handle updates and deletions without the need for expensive data compaction. Instead of maintaining complex metadata for each block, Gridstore tracks usage with a bitmask. The Gaps Layer helps optimize update handling. It provides a higher-level view of block availability, splitting the block into regions and keeping track of the largest contiguous free space. The layered approach ensures space optimization Gridstore does not waste space and data integrity is maintained.
- Qdrant relies on a Write-Ahead Log (WAL) to avoid inconsistencies with data integrity. Qdrant ensures that when committing an operation, it has recorded it in the WAL, and if a crash happens before all updates are flushed, the system can safely replay operations from the log. Data is updated lazily; Gridstore prioritizes the most critical part and writes changes in a buffer, only finalizing updates when explicitly requested, ensuring data is not accidentally marked as deleted before being safely persisted.
- To prove Gridstore's durability while handling heavy workloads, Qdrant built Crasher that tests Qdrant's data integrity limits by brutally killing and restarting it. Crasher continuously writes random data and then randomly crashes Qdrant, and Qdrant replays its Write-Ahead Log (WAL). If there is any data loss or a corrupt payload value, it is highlighted. Gridstore performs twice as fast as RocksDB and more reliably under heavy workloads and unexpected failures despite being slightly larger in size.
- Gridstore API remains tightly integrated with Qdrant, and as such, has not been released as a standalone crate. Qdrant will consider publishing Gridstore as a contribution to the community in the future. Gridstore represents an advancement for managing key-value storage needs. Qdrant achieved faster, more reliable data ingestion while maintaining data integrity, making it a win for them.
Read Full Article
7 Likes
For uninterrupted reading, download the app