Two and a half years ago, I built an open-source project called Indexify during evenings and weekends before starting Tensorlake. It was a compute engine for building a serverless durable workflow engine for data-intensive applications. If you had a million PDFs and wanted to OCR and index all of them, you’d reach for something like this.
What remains today is the underlying fast VM scheduler that ran lowe latency functions in MicroVMs. It has been powering Tensorlake’s sandbox platform and runs hundreds of thousands of sandboxes every day. We did Computer SDK's 100k challenge a couple of months back - burst from 0 to 100k sandboxes in 20 seconds on top of this service.
As we scaled over the last two months by 20–30x, we built a new scheduler from the ground up to address some of the scaling challenges and requirement to be highly available even during partial failures. It’s unlike anything in the industry: the life cycle of sandboxes and clusters is still represented as replicated state machines, but it’s heavily sharded, with very little global ordering.
Instead of building a specialized database using Raft, we now use FoundationDB to store the WAL, and use watches to replicate that WAL across scheduler replicas. Thanks to coding agents, we have been able to build a simulator which injects all kinds of failures into the system to prove the changes we make preserve reliability, speed and correctness.
We are going to push this to production next week, we are hoping it would power the next chapter of the company. Long live Indexify!