We leveraged two amazing open source projects when building SmithDB.
One is
@ApacheDataFusio: an extensible Rust based query engine. We built custom execution plans specifically tuned for our workloads and storage backend, and DataFusion made it straightforward to plumb everything together.
The other is
@vortexdotdev: an extensible file format that allows you to build custom layouts with specific encoding and chunking strategies for different columns.
I would highly recommend checking out both of these projects if you're interested in modern data systems.