My Quarterly System Health Check-In: Beyond the Dashboard
blog.nilenso.comInteresting set of points; the intent to move beyond sterile dashboards and engage in deeper, more meaningful conversations about system health is very welcome, especially in a time when most leaders don't bother to read about Goodhart's law on metrics.
But still, I spot a few points of concern.
- While experienced engineers develop valuable intuition, this can also be a source of significant bias. An engineer's "feeling" might be influenced by their personal comfort with a particular technology, their resistance to change, or their own role in creating the system in question (the "IKEA effect"). Over-relying on intuition can lead to subjective decision-making that isn't backed by evidence.
- What is "simple" for a senior engineer with years of context might be overwhelmingly complex for a new team member. Furthermore, some business domains are inherently complex, and attempting to impose a simplistic model can lead to a system that fails to capture the necessary nuance and ultimately creates more problems.
- Informal discussions can be dominated by the loudest voices, the most senior people in the room, or those with the most social capital. Junior engineers or those with dissenting opinions may not feel comfortable speaking up, leading to a skewed and incomplete picture of the system's health. A more formal "safe space" approach might help here, to increase psychological safety perception of participants, for a better discussion.
- For large, legacy systems that have been in production for years, questions like "Can we explain the system's responsibility in plain English, within 5 minutes?" or "Do simple modifications you expect in hours, take many days?" can be demoralizing rather than constructive. They might highlight known, intractable problems without offering a clear path forward, leading to shame, anxiety and frustration.
Hey, thanks for reading and commenting!
1. I agree, numbers are important, and these intuitions and feelings should be backed by numbers. In the post too, I suggest looking at dashboards during such discussions.
2. My definition of simplicity is largely based on Rich Hickey's talk, I would recommend it if you haven't seen it. I think it's possible to be somewhat objective about simplicity. If something is overwhelmingly complex to a junior, ideally a senior engineer is able to appreciate that complexity.
3. Yeah, the loudest voice problem exists, like with any in-person discussion ig. Keeping discussions on slack / notion helps side-step it. Discussion rules with timers, going around the room, anonymous comments, etc can also help.
4. A complex legacy codebase will and should fail the simplicity test, at least wrt a new engineer's experience. And it would serve the team well to accept it, and try to solve for it. Ruminating on any problem without moving towards a solution is frustrating, and can be demoralising, yes. And providing direction and creating momentum in that direction is a leader's job. In this blog post, I only offer questions, not answers :p.