I had a plumber over the other day. I was worried that my water service line might be leaking. There was a wet patch in the yard and I noticed that one of my sink’s water pressure seemed to be lower than usual. If the line had a pinhole leak in it, it could easily burst into a multi-thousand dollar flood on my hands. The joys of home ownership.
The plumber listened to my concerns and then inspected a few things: the piping where the service line enters my home, the pressure regulator, the sink in question, and even my water heater.
Afterwards, he told me it’s probably not as bad as I feared, but that I should sit down.
He produced a tablet and turned it to face me.
“Here are your options,” he started, indicating towards a group of boxes on the screen.
“First we have the ‘platinum package’. This includes replacing your pressure regulator with one that is set correctly (apparently pressure regulators can only be set once?), as well as a new water heater, a water softening system, a reverse-osmosis system, an electronic descaler, and a replacement of all your faucet aerators.”
The price was conveniently listed below: six easy monthly payments of $2,500 ($2400 if I was a “member”).
He patiently answered all of my questions about this package, although it was surprisingly hard to get an answer to “Do you think I really need all this?”
From there, we went down the line through the other options. Each cost a little less than the previous, included fewer products, and were more cleverly named than airfare classes; there was ‘premium’ followed by ‘standard’, ‘economy plus’, ‘economy’, and finally ‘band-aid’.
I asked more questions, and I was eventually able to negotiate my own “package” consisting of just what I thought I needed. For closure, no, my water service line was not leaking (probably).
And that’s how you make product give a shit about your architecture proposals.
You are the plumber
Your role as a software engineer is to play the plumber to product. The reality of the world is that product holds the money and software development is seen as a cost center to be minimized towards zero. You are selling to them.
Let’s walk through an example.
How this works
Your team successfully launched a new product for your company. It’s awesome at what it does, customers love it so far, but it’s still fairly immature.
Product comes to you and says “What our customers need next is the ability to generate reports.”
Now, having been intimately involved in the development of the product, you know how the data is stored. It’s a relational database. It’s fantastic at CRUD operations. It can perform some basic aggregation across related tables, but otherwise it’s ill-suited for something like “reporting”. Besides, product tells you that customers would love if their reports could be sorted and filtered based on customer addresses, but customer address information is actually stored in a completely separate system. And this system calls into yours, not vice-versa. And…, and…, and…
Before you freak out on product and go into all the reasons you can’t do what they want, take a step back and consider:
- Product doesn’t give a shit about how your data is stored. Product cares about products
- Product is looking for a result, not a distributed system design interview whiteboard session
- Product is human (for now), and they understand that there are engineering considerations. That’s why they came to you in the first place. They came to you for a negotation (more on this in a minute)
In other words, product doesn’t give a shit about your architecture proposal. Yet.
Channel your inner plumber.
(Aside, you’ve earned this “problem”. You designed the system correctly to do what it needed to do in order to be a successful product. The natural order of successful products is to evolve beyond their original infrastructure.)
So here’s what you do instead of stuttering through an explanation of “indexes”, “table joins”, and “cardinality”:
You show them the “platinum package”.
This means that you gather up all the information you need to give product exactly what they want, and then you come back to them with an estimate: six easy monthly payments of $2500. Or, rather, you say “One full time mid-level engineer’s time for 6 months on our team, plus one full time engineer’s time from the Infrastructure team.”
Taken aback at such a large estimate (because they were hoping, pretty please, that it would be a single two week sprint. After all, all the data they need is “already in our system”, right?), product utters a single-word question:
“Why?”
And that is how you get product to give a shit about your architecture proposal. Suddenly they want to know all about it because they know they can’t afford a full year’s worth of engineering work.
Now you can (gently) talk to them about the difference between online transaction processing systems (OLTP) and online analysis processing systems (OLAP).
Each thing that needs to be done is a line item in your ‘platinum package’ invoice:
- Define an ETL process to flatten and export our data to Snowflake
- If you want, you can further break this down into the batch processes that happen, and when, where, and how often they happen, which informs data freshness guarantees
- Similarly define a process to flatten and export the “customer data” from its system
- Provision a new Snowflake instance
- Front-end and backend work to provide the customer a way to specify what data they want and how
- Translating the customer’s request into Snowflake and back
- etc. (Saving reports to a document store, emailing them, running them on a schedule, etc. etc. etc.)
Be nice
During the discussion, you will patiently go into as much detail as product wants about specific line items. You will explain acronyms and technologies. You will explain costs of each to the best of your ability.
And then they can turn around and ask you “Do you really think I need all this?”.
Ah who am I kidding? Product doesn’t give a flying fuck about your opinion, you dirty code monkey. They’ll ask themselves “Do I really need all this?”
And the answer might actually be “yes”.
Sure, it’s hard, and expensive, and time-consuming to do, but that’s how businesses make money; by doing the hard, expensive, time-consuming thing for their customers so their customers don’t have to. Granted, that’s only if your company hasn’t yet started confusing “build something our customers need” with “build something that conveniently fits into a 2 week sprint” (and that’s a big “if”).
I’m not gonna hire a plumber just to wipe the gunk off my sink faucet. That shit is $87 just for them to ring your doorbell. Anyway, I digress.
It’s a negotiation
Understandably, product will want to deliver something to their customers sooner than six months, so they’ll reply with things like “What if we put off saving and emailing reports until later?” or “I didn’t realize it would take that long to get these running on a schedule and uploaded to the cloud”, or my favorite, “You are so smart and good looking and totally not a slob that eats lunch at their desk so often that their keyboard has accumulated a mass of crumbs to the point where the shift key sometimes can’t go all the way down, can’t we do everything in the platinum package in two weeks instead of six months?”
A surprising number of developers fall for that last one.
Like, most of them.
Sure, you could “just” murder your database with table-scanning queries that join every single table and hope that you’ve provisioned a beefy enough machine to handle the load “for now”. Just like your plumber could “just” fix a pipe leaking on the floor by shoving a bucket under it and telling you to empty it every week.
But you gotta take a stand on quality somewhere.
Developers fall into the same trap of thinking in terms of whatever length of time their organization parcels out work into: two week “sprints”, six week “cycles”, what have you. And product is telling them the scope can’t budge. And you know sure as shit you ain’t getting any more team members to handle the load. What’s the only thing that can give at this point? Quality. That’s the iron triangle of software development, and you’re its latest victim.
So let me squash the idea of sacrificing quality real quick by asking you a question:
What’s worse, delivering something a customer actually hates, or delivering nothing at all?
From painful experience, I can tell you that you’ll lose customers if you choose the former. And the former happens when you sacrifice quality. You don’t want a goddamn bucket under your pipe that you have to empty, you want a new pipe.
Grow a spine and find a level of quality that you will not compromise on. For example, my plumbers won’t install any off-the-shelf parts. They’ll only install products that they supply themselves because they are confident in the quality. At least I charitably believe that instead of the alternative which is that these products are expensive as heck and yet another way to gouge me on the margins.
My point is that quality has some fixed minimum and therefore scope and time are what budges instead. If you make a big enough of a stink you might even get more budget and a bigger team, but if you manage to accomplish that please tell me how.
Now back to the show.
It’s a negotiation, redux
After you’ve decided to be a big strong boy that takes a stand on some bare modicum of quality, we can start the negotiations.
When product says things like
“What if we put off saving and emailing reports until later?”
or
“What if we dropped running reports on a schedule?”
This is when you start bringing out your ‘standard’ and ‘economy plus’ packages etc. Because you’ve already gone to the trouble of treating these things as line items in an invoice (read: estimated their effort) you can tell product how much their chosen package will cost. You can even give them estimates on when you can deliver these line items, allowing your organization to plan over a longer time horizon with deliverable dates along the way.
It gets interesting when product goes a little off-script and says something like
“What part of the desired scope can you deliver if I don’t want to pay for an OLAP system?”
Then you can get creative. You are negotiating with product on the two facets of the project that will generally change (mentioned in my rant above): scope, and time.
If I were jaded (I am), I’d say the only thing that really gets negotiated is scope because for some reason every shop out there has decided that an entire project needs to fit within the arbitrary window of time that constitutes whatever definition of “agile” they’re currently following.
If you’re reading this thinking, “No way, management at my company take a longer term view of things in favor of delivering best-in-class products”, then I implore you to include that information in a letter you send to:
His Holiness
Casa Santa Marta
00120 Vatican City
So they can be canonized as the first living saint.
In many ways, you are simply helping product to ruthlessly prioritize a list of work so that the team can deliver the highest return-on-investment (ROI) within the confines of the fixed time, and budget points of the iron triangle. If there’s enough good stuff left on the table at the end, you might have accidentally planned ahead for your next project for once.
Turning it around
If you’re doing your job right, there’s plenty of crap that YOU want to do, but so far we’ve only talked about making the people with the money give it to you when THEY want to do something.
So how do you get product to give a shit about your architecture proposal when the tables are turned?
Well, your local insulation company will gladly come to you, for free, and spend an hour with really expensive thermal imaging gear to find walls that need more insulation. Free because they know that your home’s “builder grade” insulation made of bits of drywall and discarded Monster Energy cans is literally garbage. They’ll tell you that you’ll save $50 a month in heating and cooling costs if only you could give them $5,000.00 now.
You’ll quickly do the math and ask yourself a few questions:
- am I really that uncomfortable?
- do I plan on staying at this home for roughly another 8 years (the time it takes before I’ll recover the investment), and
- if I were going to invest five grand, would it be better spent elsewhere?
Wow suddenly you’re the product team. Go take a cold shower.
The equation doesn’t change
Nothing has changed here except who came up with the idea. Be the insulation company. You still need to come up with line items in an invoice as before. You still need to negotiate time and scope. You still need to make a compelling argument about ROI. Here’s what’s not going to get your proposal greenlit:
“I want to add a couple more tests to this code the intern wrote last summer”
How the fuck is your company supposed to translate that into ROI? Try this instead:
“I noticed that (business critical feature) is under-tested, so I would classify it as high risk of having bugs in it that would be showstoppers. An outage here could cost us big $$$ not just to fix it, but also in customer churn and reputation loss.”
And even if your project gets greenlit, it may be after you make some concessions like the project can’t go on for longer than a week, or that you can’t bring in anyone else on it (that’s the negotiation part).
You have some petty cash to spend
The last point I want to make is that you probably have a little free time to do what you want at your job outside of feature work and chores like keeping third party libraries updated. Google called this “20% time” in the early 2000s (and their engineers called it “120% time”).
For the last time, the equation doesn’t change. It’s just that the same person (you) holds the money and proposes the work. You may not need to write out your line items on paper (although it would certainly help); most devs somehow keep that in their head alongside all the movie quotes they endlessly repeat in Slack.
Don’t feel obligated to immediately spend your allowance on small things like linting fixes. You can invest it each week towards something bigger like a pub/sub system so that you can pull a new microservice out of your monolith. Just make sure that the rest of the team is onboard with the bigger changes.
Summing things up
When you are elaborating work for product’s next brilliant feature that less than 1% of customers will ever use, elaborate how to do it right. The cost of time, scope, and budget fall out from there. Let product decide if they want to make that investment. Negotiate with your product team on what you can deliver and when, and never back down on quality. Easier said than done.
Harness your inner plumber. Sell that platinum package. It will make dealing with all the shit everyday a little easier.
Until next time.