Why does so much controversy and disagreement persist within the software development field? Why don't we converge on a favourite language, database, or design pattern?
Of course, you can think of plenty of reasons. No two software projects are identical, and some approaches are valid in some circumstances and not others. There's a cost learning a new tool when your current one seems good enough, and similarly, a cost of rewriting existing code in a new language. Humans are tribal and like to stick with their own. And so on.
But I want to discuss another consideration in this post, related to the "Peak End Rule".
Peak end Rule
The Peak End Rule states that humans' recollection of pain (or pleasure) is biased in the following ways:
- towards the very end of the experience, and
- is not much affected by the duration of the experience.
It's a surprising result: you might think a long holiday will be remembered with more affection than a short one, but the human brain tends to remember by reference to prototypical moments, especially moments at the beginning and end.
The original psychology experiment investigated participants' memories of a deliberately created discomfort. From Wikipedia:
The first trial had subjects submerge a hand in 14 °C water for 60 seconds. The second trial had subjects submerge the other hand in 14 °C water for 60 seconds, but then keep their hand submerged for an additional 30 seconds, during which the temperature was raised to 15 °C. Subjects were then offered the option of which trial to repeat ... subjects were more willing to repeat the second trial, despite a prolonged exposure to uncomfortable temperatures. Kahneman et al. concluded that "subjects chose the long trial simply because they liked the memory of it better than the alternative (or disliked it less).
So even though the longer trial was overall a longer unpleasant experience, the memory of the experience at the end dominated.
The Peak End Rule and learning
Feedback is well known to be important in learning. In agile software projects, we try to get feedback as quickly as possible and adapt our processes accordingly - we learn from it.
Sometimes though, you can't immediately get feedback on a design decision or a technology choice. Whether or not it was the right thing to do, only becomes apparent over a long period of time.
The end of (most) projects
Now think about the typical software project that ends up in too much technical debt. I think most projects end up like this, if we're honest. Towards the end, it feels like we can't get anything done; we're unhappy.
We're not necessarily thinking about the "good times": about the number of features that were developed along the way. A project that delivered 5 features before collapsing, might be remembered in the same way as one that delivered 50, if it had the same unhappy ending.
So even though we might have actually made good choices and delivered more features overall, we don't learn that. Our memory is too similar to the memory of the project where we got less done.
Remembering the good times too
What can be done about this? I have a few suggestions.
Keep records of what you spend your time doing. It doesn't have to be super-accurate, but imagine if you could analyse exactly how much time you'd spent wresting with shiny tech X, over a six month period. Is that amount of time what you expected when you chose it?
Take regular evaluation checkpoints throughout the lifetime of the project. Don't wait until the end. Then re-read the notes you made at each of them.
Celebrate experimentation and encourage discussion of failures. If this becomes part of your team's culture, people will not be afraid to try new things.
These techniques might help boost the salience of the overall experience in your memory, rather than the end of the experience.