For my latest quixotic project I decided to develop it in an unstable, “feature-first” manner. The exact opposite of how I usually develop software.
In the past I’ve usually tried to write code in a sort of “bulletproof” style, with every change and feature being over-engineered, over-tested, and fully reviewed before merging to main. Very similar to how TigerBeetle has been publicly working with their Tiger Style.
I explicitly chose to go a different route in developing Eidetica pre-1.0 and I want to lay out my reasons why.
This only applies to the in-development/unstable phase pre-1.0. As I approach and reach 1.0 I’ll ramp up my stability expectations to make it into a fully stable, hopefully bulletproof backend.
TL;DR - Eidetica will be unstable until the primary features are done, but not a moment longer.
Eidetica Background Link to heading
Eidetica is a decentralized database that is my solution for writing local-first software.
It’s intended to be a “batteries-included” backend for apps that want to be local-first. Store your data/files inside of the Eidetica data model and it will handle synchronization to all client devices. Users will have their data always synced to their local devices in the background, making both always-on network connections and reliance on central servers unnecessary. And by having user data always stored and editable locally there’s no slow-down in the user experience waiting for network roundtrips.
It’s peer-to-peer, optionally end-to-end encrypted, all that good stuff.
It’s also currently entirely unstable. I have a big scary note in my README that everything can break without warning and with no migrations supported.
Why Feature-First Link to heading
Honestly it makes me uncomfortable to develop software this way. Working within an unstable codebase feels icky but it has a lot of benefits for early development.
1. Development Velocity Link to heading
I am building this myself, because I want to get the core designs down first.
But that also means there is only 1 developer, with 0 funding except for my own limited ‘fun’ budget. (I’m FIRE’d, which means I’m rich in time but have a limited budget).
By holding off on stability as long as possible I can avoid wasting time on that when there are very little/zero actual users.
2. Chicken/Egg Problem Link to heading
Without stability there can’t be users. But without users there’s no point in being stable.
However, there also aren’t users without sufficient features. There’s no point in paying for stability until you have to.
3. Don’t Know What I Don’t Know Link to heading
It’s hard to change core configurations while trying to maintain stability.
And the truth is I don’t know the right way to build this.
Sort of.
I gave up on trying to design the whole thing correctly from scratch before building it. It’s too large to fully model without existing code, and I’m trying to avoid making a wrong decision in the core that I will have to support forever.
Because of how Eidetica is built it keeps the entire immutable history around, so if there is a mistake in the design it will likely live forever.
And I know there are mistakes! I’ve already fixed a bunch of them!
Instead of trying to figure all those out from the beginning I am building all of my primary hero features before stabilizing. Focusing on the ones where I’m unsure how or if they can fit into the existing strategy.
4. All Features are Required Link to heading
The subset of Eidetica that needs to be built and stable in order for apps to use it is actually quite large, larger than I thought back when I started. And that number only grows larger when I add in the features on the roadmap that may have an impact on the core.
Here’s an incomplete list in no particular order:
- Built-in CRDTs
- Bring your own CRDTs
- Efficient state caching
- Large Table support
- Multiple Hash function support
- Multiple private Key types support
- P2P synchronization of the Database
- P2P object storage synchronization
- Basic Encryption
- Good Encryption
- Database Management facilities
- Background service
- Local client-service communication
- History browsing
- Authentication and Access Management
- Administration/Moderation
- Sparse Checkouts
- Fast Initial Sync
- Bootstrapping
And probably more I can’t think of right now.
Some of those are already built, some of those I have solid designs for how they will work and can wait for 1.0, and some of which I’m completely unsure of.
I’m trying to target things that are definitely essential, and then have reasonable plans or ideas of how others can fit into the existing design without breaking it.
How to Stabilize Link to heading
Since I’m building so much unstably I will eventually need to stabilize it. For that I’m doing 2 things very intentionally.
1. Layered Design Link to heading
The majority of the features are being built on top of Eidetica itself.
- Authentication/key management reuses database administration primitives
- Instance management, user data, and synchronization info are all being stored in Eidetica databases even though they don’t need history or synchronization
- My own features all use my own simple CRDTs
- Etc.
As much as possible it’s being built in this sort of “nesting doll” style design where each layer builds upon the one below it. As I build and test more features, the lower layers of this stack stabilize with bug fixes and cleanups as I build features on top of them and find more issues.
Basically, Eidetica itself is the first consumer of the Eidetica primitives and design as much as possible.
2. Extensive Testing Link to heading
My philosophy is that compute is cheap and dev time is not, and that still applies even though I don’t have a salary.
That’s not to say that I don’t care about efficiency in my code, far from it actually, but rather that CI, testing, and other development support things can and should use computing power to save dev time. If the goal is for the final built code to be run at scale then having a few machines constantly making sure it’s fully tested and fuzzed should be the default.
TigerBeetle, for example, constantly uses 100 cores just to run their simulation testing infrastructure. Eventually I’d like to emulate that for Eidetica.
I like having lots of test coverage via unit tests and integration testing, and I plan to also build out fuzz and simulation testing once I approach a more stable base.
Conclusion Link to heading
So for Eidetica it seems to make the most sense to build out this unstable foundation that is underpinned by solid Computer Science theory and research, then transition my efforts into large amounts of testing/stabilizing once the essential features are in place.
Instead of trying to determine the exact right path from the beginning, this route of trying to build the whole system in an unstable manner has been working remarkably well.
Well, maybe. It’s certainly allowed for a very high development velocity right now, and I guess time will tell if it leads to a solid, stable design in the future. I’m more comfortable and experienced with the ‘stable’ development style so I am reasonably confident that that will go fine.
So in short: Eidetica will be unstable for as long as is reasonable, but not a moment longer.