At
@rocicorp we've always had a design doc tradition (inherited from Google I suppose).
Since we are a distributed team, we would do design docs up front that are fairly detailed (typically 3-5 printed pages and taking maybe a week or two of work for each new major component.
We'd discuss and iterate them as a team, then work proceeds independently - w/o blocking code review - and we stitch together at the end.
This might sound crazy to many ears, but when you're working on databases and sync engines and things like this - that are pretty subtle - we have found it makes sense to think about things at a high level first, before code.
This always worked on our prior projects.
For
@rocicorp_zero, we found this approach breaking down. The design doc system is kind of predicated on the idea that the people reviewing the docs have a big picture understanding of the whole system and can understand the new component in relation to that.
But we didn't have this with Zero. None of us did. It was too big a project to fit in any of our heads at once early on. And we found we couldn't design (or review the design of) piece A without understanding piece B, C, and D first.
We also found it was difficult to design some parts "on paper" without the help of the typechecker and the deeper understanding that comes from real code, a keyboard, and unit tests.
We ended up – not really purposefully, but sort of out of desperation - writing several versions of the IVM pipeline as prototype/throwaway code. Several of us took a swing at it 😂. And through this kind of iterative/explorative process we got more of an intuitive feel for the system.
Once we had this, we started working on real code (ironically w/o a design doc at all – there still isn't one for ivm). But we had to accept that some parts of the system we didn't know exactly how to do. We had to take a sort of leap of faith that we'd figure them out when we got there.