Driving engineers to an arbitrary date is a value destroying mistake (2020)
iism.orgThe problem that the article doesn't address is that users don't actually seem to mind using terrible software so long as it solves the problem they face better than not using it. I could list literally hundreds of half-assed, broken, bloated applications that I've encountered in the past 25 years that have done very well simply because they kind of solve a problem a bit for the user.
Pushing out something completely broken that doesn't do what it's supposed to is definitely not going to work (duh!). Pushing out an app that solves the problem of managing shopping lists that has a bug where it doesn't work given a particular set of circumstances will still lead to many people using it if the users don't have any alternatives and it's better than using a piece of paper.
Software quality is important to companies because it means that they can spend more time building features instead of fighting fires, and because low quality represents a threat that a competitor could launch a better, less buggy app. Users mostly don't care so long as the app works well enough to do what they need it to do (but they're not dumb, they'll still pick the least buggy option if there are alternatives..).
A high level of quality in software is not important unless you're entering an already well-served market. I wish it was.
Personally, I write software that I consider extremely high-quality. The folks that use it, seem to agree. It isn't eye-candy fancy, but it works very well, in a not-in-your-face manner.
The idea is that it does what it says on the tin, without fanfare, robustly, usably, accessibly, localizably, and dependably; providing a user experience that gets out of the way of the user, in a manner that does not surprise the user (even "good" surprises can be an issue. Boring software can be just what the doctor ordered).
In my book, that's the definition of "quality."
I'm working on an application that has been over a year in the making. Its functionality is something that I could have popped out in a month, but making sure of the Quality of the app has necessitated that I spend a great deal more time, "polishing the fenders."
If this were a commercial app (it isn't), then it would have been unbearably expensive for a startup.
I tend to write test harnesses in a day or two, that have similar levels of functionality to this application.
High Quality is significantly more expensive than even "decent, but lesser" quality.
You worked on that app yourself alone, right? If so, I believe your app belongs to a different plane of existence. Programming (one programmer working alone without too much money/time pressure imposed by others) is quiet different from software engineering (multiple programmers, stakeholders, designers, etc., with time and money constraints). Everyone loves programming (and so doing programming usually leads to high-quality and useful products), almost everyone hates software engineering.
I don't like this definition of Software Engineering that necessitates a team of Other People. Perhaps I'm outright wrong that such a definition just shouldn't be. Even in that case, I just don't like it. :)
If you broaden the definition of Other People to "you, 6 months from now" and "you, 6 months ago", then many SE principles still apply. Where SE principles don't apply is toy, one off scripts that you just hack together to get a single thing done and require no maintenance like a lot of stuff in science.
I'm the poor schlub that usually needs to maintain the code that I write.
I also pretty much never get questions about the code that I pass on to others.
I write about my process here: https://littlegreenviper.com/miscellany/leaving-a-legacy/
(Long screed. Few read it).
Absolutely true. Not even 6 months needed.
When I do stuff for myself I apply the same principles I apply at work. It's insane how easy it is to change stuff later on. Didn't think about this use case before but now you do? Because I have properly maintainable code that is readable its very easy to change and changes are only needed in one place instead of all over the place. Knowledge of the right thing is kept in the right place instead of implicit knowledge all over the code etc.
It also helps to have 'one team be responsible for each service' instead of 'everyone can work on everything'. It's insane how fast you can move if you know the code well and it's maintainable.
Strangely enough, software engineering is mostly about humans rather than about software. Take any side project of yours (no matter if it has been hacked together or if the best practices out there have been applied): add pressure to make money out of it, pressure to deliver it at a given date and a bunch of individuals you must collaborate with to push the project live, and right there you get SE.
I would say a lot of this confusion comes from the word engineer. Building software is not like building a bridge. You can't design it first and then go and build it.
So I would say engineer is a poor term, but also the only one we've got for now.
As I see it, it is more nuanced. When engineer designs a bridge she does a very similar work to a programmer. Programing it is like designing a very complex bridge without constructing it really, because when you finished your design it had been built already.
So it is not exactly like you've said:
> You can't design it first and then go and build it.
You can design, but you cannot build.
A process of building by a design can be paralleled with deploying software -- suddenly there is a hairy real world, not all the hair was considered at the design phase, and either we hack around existing software (i.e. design plans), or call a programmer to redesign.
No it's not like building a bridge. It's like thinking you're building a bridge but then the client decides he needs it to be moveable so then you have to give it legs so it can walk. Then the client's manager decides he wants it to be able to get to the moon so you then have to strap rockets and parachutes to it. Then after it gets installed and someone finally speaks to the users you realise all they needed was a canoe.
> You can't design it first and then go and build it.
You can't design it in full in one go, but you can design it and then incrementally update said design. Sadly many (companies) do not. But you can define the problem(s), the scope, the scale, and then design a solution appropriately to meet those needs (for a defined period of time). That's what distinguishes software engineering from hacking. They both have their place. Many companies claim to do the former but are mostly doing the latter. Software is still early in its life and as various kinds of system designs stabilize, so will the formalizations around what it means to be a software developer. Reading a book like Designing Data Intensive Application's you can't help but see those formalized topics budding.
It's exactly like building a bridge... if nobody never built a bridge before.
The only reason you can design a bridge beforehand is because (millions?) bridges have been built before so you can apply the lessons learned. Even if your bridge is "unusual", it will still be similar enough to older bridges so you don't have to invent the vast majority from scratch.
Other kinds of engineering don't have the luxory of leaning on the prior experience so much, simply because there is less of it. SpaceX's reusable rocket could not have been be fully designed before built, simply because nobody built a reusable rocket before. But it could be done through iteration, which is just another name for experimentation.
Software tends to be less like bridges and more like rockets... all of which falls within the spectrum of "engineering".
In Canada, engineer is a protected title. There are software engineers and you get professors who stress to you the importance of testing and safety, and there are safety and ethical requirements. It’s not very strongly enforced though, lots of people here get away with using the title
Yes, I personally find developer or programmer more accurate, although neither is without its own connotations and baggage.
As a designer I’m a bit embarrassed by the trend to describe ourselves as product designers. To me a product designer makes chairs and coffee tables.
I mean, I'm and audio engineer, a fire protection engineer by degree, and a software engineer. Your bridge building analogy doesnt hold up, there is always changes and refinements of guestimations.
Hmm, 'developer' has a cheerful kind of vagueness about it - it comes close to describing the iterative process of starting to write something one way, then realizing your original approach wasn't quite right and rewriting it. At least, I don't think mechanical engineers do that too often...
Yes, I've been writing it alone, but my "alone" work is at a different level, from what many folks do. Approaching every bit of work I do as "ship," is one of my oddities. I have as much fun, shipping, as I do writing.
I consider everything I do, "engineering." Been doing that, all my adult life.
Feel free to look at the stuff I do (I link to it in my HN profile). The app I'm working on isn't there (yet), but a number of its components are. It's still "under wraps."
I hear you, but this problem becomes exponentially harder as your engineering org grows.
I witness on a daily basis PRs that have no body getting merged with absolutely zero comments and a blanket approval as long as it passes our (broken) CI pipeline. I witness obvious poor quality in the code, but engineers want to seem like they are working and will just blanket approve PRs, while i'm in the middle of writing up my code review denying the PR.
If you are a developer on a team and want your codebase to be high quality, you end up no longer writing the code and instead spend all of your time gatekeeping via code reviews. This leads to burn out.
Please come work at Google! This place has the best overall code quality I have ever seen. That's not to say it's always great or bug-free, but it is almost always well maintained. There is very little dead code. Giving and receiving high-quality code reviews is the norm.
That’s one company that would almost certainly not have me, but I very much appreciate the thought.
Apple also displays extremely high quality code, in the areas that are exposed to the public.
I have not seen too much Adobe code, but I’m told you can eat off the Photoshop codebase.
For myself, I need to keep my scope fairly humble, but I get great joy from it.
It sounds like a gratifying environment. Good show!
Hey, I found your email on another thread.. I will reach out!
Yeah. It can be a real challenge, ensuring quality on a team.
The obvious answer, is to hire experienced, skilled, capable engineers, and instill in them, the same reverence for Quality that you have.
Like I said, Quality is expensive. Very few companies like to pay the premium.
This does not work at scale. My company does pay the premium and a lot more.
It does, but it ain't easy. I worked for a company that did it at scale.
I wonder if the nuance being identified by the other commenters might be the tragedy of the commons that can play out in collaborative settings. I'm right there with you, btw. Mine is also a non-sexy, non-commercial app and I'm happily a one man band. But a younger me, a junior member of a team, under pressure without close supervision would probably commit a certain amount of half baked spaghetti and be part of the governance problem.
Yup.
It's not exactly a "one-man show," but I'm the chief architect, and the only one developing one of the three servers the app uses (I was also the original architect for another server, that is now being run by a different open-source team), I am also the only one developing the native iOS application. I may be writing some adjunct apps, once the main one has been released (like Watch, Mac and TV apps).
But we're a team. It's a 501(c)(3), with a mission to Serve a specific demographic. We have the advantage of being intimately familiar with the demographic. So far, we haven't had to shell out much. If they decide to write an Android version, then it may take some extra dosh. The good news is, the app is in "constant ship" state, so asking for funding is fairly straightforward. We just need to loop the person into the TestFlight group, and Bjørn Stronginthearm is your uncle.
It's easy to make high quality software as a team of one - you never have to compromise, after all. You are basically a god of the project. When working on a large team though you have to make all sorts of trade offs at all sorts of levels and it's way harder to produce high quality software because you are no longer a god that controls every single aspect of the project.
I don't buy it. You don't have to own the _ENTIRE_ application to be a developer of one. I've seen many developers that were solely responsible for a specific feature and shipped what I'd consider poor quality.
Large software project, whether you're a developer of one or a team (and probably more commonly occurs in a team since it's large) will have warts. It's harder to manage complexity as the projects size increases since it accumulates and it accumulates fast.
> I'm working on an application that has been over a year in the making. Its functionality is something that I could have popped out in a month, but making sure of the Quality of the app has necessitated that I spend a great deal more time, "polishing the fenders."
What happens if you release it and it turns out that nobody likes it? I'd have preferred to spend a month releasing a lower-quality app, and then spent 11 months improving it (while people use it and I learn more about what they want) rather than take a year releasing a high-quality app nobody wants to use.
This assumes you aren't just building it for fun, though, in which case building it is the point, and you don't even need to release it afterwards.
You are correct, and that's the classic rationale for "MVP."
But I also follow a development methodology that I call "paving the bare spots."[0] It adjusts the design, as the project progresses. Not for the faint of heart. It means that I may toss out a month's work, because it does not fit the user experience (which is under constant revision). We have also made a couple of major pivots, during the project, to fit new realities.
I have probably tossed out three months' worth of work, as the project has progressed. Happy to do so. The best code, is the code I don't write.
The nice thing is, is that the product is constantly at "ship" Quality. Makes demos, user testing, and begging for money, easier.
[0] https://littlegreenviper.com/miscellany/the-road-most-travel...
Professional developers should strive to write the highest quality code they can given the constraints of the project. That's pretty much a given. I'm only arguing that customers don't seem to care, not that we shouldn't care.
Exactly.
I read a book, where one of the characters is a smith. It has this exchange, between him, and another character:
"Always do the very best job you can," he said on another occasion as he put a last few finishing touches with a file on the metal parts of a wagon tongue he was repairing. "But that piece goes underneath," Garion said. "No one will ever see it." "But I know it's there," Durnik said, still smoothing the metal. "If it isn't done as well as I can do it, I'll be ashamed every time I see this wagon go by -and I'll see the wagon every day.”
(from Longfellow, The Builders, 1850)In the elder days of Art, Builders wrought with greatest care Each minute and unseen part; For the Gods see everywhere.I’m not sure to which extent I agree with this piece’s medieval outlook of seeking and expecting perfection in the past, not in the future, but it doesn’t detract from the quality of this piece of writing. (Same for Tolkien, for example.) (And the poem actually talks about improving on the past, not venerating it; citing this part in isolation is a bit misleading.) (Now that I’m comparing these two quotes, the difference between “because the gods will see” and “because you’ll know it’s there” is... probably not worth overanalyzing, but at the same time intensely amusing.)
I, too, am a firm subscriber to the Goodman Durnik school of development. With an occasional bring-a-flaming-sword-into-a-meeting-room-to-emphasize-your-point twist.
One reason I like this kind of work is that it implies when someone spent that much effort on an irrelevant detail they probably paid attention to important parts as well. Not always the case but yeah.
"We are what we repeatedly do. Excellence, then, is not an act, but a habit."
- [Probably erroneously] ascribed to Aristotle
But naysayers will say we are "bikeshedding."
Meh. Whatevs. I do things the way I do.
It’s a paraphrase, but the origin is indeed Aristotle.
I agree. I've been naive at the beginning of my career, thinking that a great product sells itself, but it's simply not true.
There's so much trash everywhere that is earning millions (book reading apps, silly chart-db software, proprietary SQL databases etc.).
I was quite shocked by the levels of disfunctionality in the software engineering sections of the company I worked at yet still, the company has huge success. We basically got requests from the product team to make something work in a day or two and even though it was completely horrible code that absolutely paralyzes us at some moment in the future, the company still reigns.
I like what Peopleware says on the matter - the market has squarely selected low quality software over high quality, but if you try to drive your teams to build low quality software you will lose because you will be unable to retain good developers.
Well, the market does not select for low quality, it just selects for an attribute that often coincides with low quality, like time to market or low cost.
The problem that the article doesn't address is that users don't actually seem to mind using terrible software so long as it solves the problem they face better than not using it.
I've been saying that for a long time, but I don't see as a problem in itself, but as a fact of life.
The problem comes because people get the wrong impression that "state of the art" is always a compliment and terrible usability is best of possible worlds.
I'm reminded of this every time I try to teach my mother anything about Android or Windows. Or even when I must deal myself with a new app with its quirky "state of the art" GUI.
Android apps dont solve problems better than anything else, they are just more available so this doesn't gel for me.
Windows and Android keep reinventing the same wheel even though (especially Windows) is basically still in "Windows 95" mode - getting fancy with a phone ui is just trash.
You are right.
Solving the right problem the wrong way is more important than solving the wrong problem the right way.
As an engineer, that has been one of the most important lessons for me in my career so far.
Or: Assuming a vacuum, solving the right problem is more important than solving it the right way.
But also, solving it the wrong way may create an opportunity to do better.
There is also a difference between buggy for the first time, which tends to cause a user to dismiss the app, versus sometimes buggy after the app is well established as useful for the user. At that point, there is a different dynamic at play for competitors, they need to be perceived a significant percentage better than the incumbent to overcome and justify the effort of the user changing. I have no sources to back that up sorry, just something observed.
Just guessing, but I suspect this is because most software is designed to extract some minimum acceptable investment per person from maximum people, and simply focus on scaling.
When software is designed to focus on extracting maximum investment per person, and focus on whaling vs scaling, then you get loot boxes. Suddenly overall software quality matters a whole lot to the user base.
> whaling vs scaling
This is a great phrase that I'll have to remember. Pretty much all enterprise sales is "whaling".
And yet, what I see is that enterprise software seems much less polished than consumer software.
My assumption is also that enterprise software contains _more_ bugs than consumer software.
That's probably why articles like this get written so often. The companies who create enterprise software seem to think they're still working on scale.
And I don't intend my loot box reference to be taken too seriously. The gaming industry isn't any better. As an obvious example, Cyberpunk 2077 was delivered as a hot mess only to meet overhyped timelines.
But look at games like Apex Legends, Fortnite, etc. They work very diligently to ensure the core gameplay is solid so streamers will provide eyeballs, driving lootbox sales. Whales provide the biggest investments there, with some individuals spending thousands of dollars for cosmetic lootboxes in what would otherwise be a free game.
Then look at games like Quake Champions which should've been successful in the same way, and completely failed because they didn't focus on making the core game tech (net code specifically) rock solid before attempting to monetize. They immediately lost the pro crowd and failed to convert the people playing Quake Live (or even the die hard Q3A players).
Enterprise software often doesn't spend enough time improving core process loops, or worse, provides too many core loops making the experience disjointed and unproductive.
Yes, the "whale" is the single huge client who pays a lot of money for the enterprise software. The client is not the users. The sales pitch for enterprise software is where the polish goes.
For Enterprise software I take auditable, stable processes and performance over a polished UI every single time. Also talking as a long time user of the poster child, SAP. Enterprise software doesn't have to be pretty, it has to do its job. And that requires a lot effort, I'd argue more effort than developing a polished consumer facing app.
I am guessing it has to do with the style of product managers in enterprise. A lot more non-technical PMs, people with BI backgrounds?
” A high level of quality in software is not important unless you're entering an already well-served market. I wish it was.”
I’d say better way to say the same thing would be ”Unless you are building something truly innovative and unique that people must use anyway no matter how bad the experience is, software quality matters.”
The thing is, 90% of software is “unique”, in the sense that it solves a specific business requirement
I guess it depends how you look at it, but regarding my point it isn’t.
For example most ecom apps/sites are surely ”unique” based on how you described it, but customers have nearly unlimited options/alternatives these days.
I was referring to enterprise internal or b2b software
The first mover advantage is huge. Once users have already chosen to use your software it will take something substantially better over an extended period of time for most to move over to a competitor that has solved it better. The risk of that happening when you are already making money and solving the problem and have most of the market is small. If you keep irritating your customers and they have a better alternative then you will slowly loose share of the market.
The cultures that tend to ship buggy software also tend to be the sort of cultures where the quality doesn't improve in response to a competitor however. But so far almost every software business has ultimately failed so being on top for a decade because you shipped some of the solution earlier works better. Customers are more than happy to buy exceptionally buggy software and games.
The first mover advantage is an illusion. Google didn’t invent search or online Email. Microsoft didn’t pioneer personal computing, spreadsheets, or word processing. Facebook wasn’t the first social network. Apple made most of it’s money from markets it entered late iPod, iPhone, and then iPad where the Newton failed. Intel wasn’t the first to build a microprocessor. IBM didn’t invent the computer.
The first mover advantage is an illusion.
I wouldn't go that far. All the cases you cite are of products that definitely classify as substantially better than what they replaced.
Before Gmail took over the market from Hotmail there had been several other companies that failed due to only being slightly better.
Yes you can beat the "first mover", but doing so is hard and requires an almost revolutionizing better product.
I am not saying it’s easy to beat them, rather first movers also generally die or move on. Atari, Electronic Controls Company, Mosaic, etc simply aren’t around any more. Sure Yahoo isn’t dead, but it’s also moved on from it’s initial success.
first movers also generally die or move on.
Sure, but before that they generally make a lot more money than the second and third mover. When all is said and done, Atari and Yahoo did much better than Colecovision and Lycos.
The risk with being the first mover is that, having easily seen off competition from the second and third mover, you become complacent and stop moving.
Many arguably most first movers simply flopped, look at social networks or MMO’s for example. The bias of only remembering successful first movers is exactly the illusion I am talking about. Generally if a slightly different business model works it’s a first mover if it fails it’s a bad idea.
Google search was two orders of magnitude better. Gmail launched with 1G storage when hotmail offered 10M. Both products launched with exponentially growing customer base too.
Despite that, hotmail still exists.
While first mover doesn't guarantee success, it's certainly an illusion.
Two orders of magnitude is a serious exaggeration. I remember swapping back and forth through multiple search engines well after Google showed up. Search index size and update frequency used to be a much larger issue for me.
Early on I swapped from yahoo mail to gmail to hotmail because gmails spam filtering wasn’t good enough. Among my fiends Gmail really won on UI not space.
Depends on where the “terrible” is. People are happy to work around UI/UX issues so long as they can get the job done (chronic ergonomic issues notwithstanding perhaps) but anything “back end” or “low level” can have catastrophic implications for mere usability. It’s funny because we often seem to ascribe higher priorities to the more visible stuff.
> users don't actually seem to mind using terrible software so long as it solves the problem they face better than not using it
This is true and, once you get over the myopic code focus that most of us shared in university, incredibly self-evident. "Users prefer to use the thing that solves their problem better than not using the thing." I mean... yes?
The real takeaway is that "software quality" means something very different to users than it does to developers. Developers want efficient, ergonomic libraries, aesthetically pleasing logical structures, and elegant software designs. That's what we call "quality". Users just want their damn problem solved, and if they can GET software like that which solves that problem, then you bet they'll buy it, but if they can't then anything which actually solves their problem is far better than doing it by hand.
This was the first big lesson I learned out of uni and it's stuck with me ever since. If you're interested in making commercial software, you're not there to solve the problems YOU think are important, you're there to solve the problems YOUR CUSTOMERS think are important.
> A high level of quality in software is not important unless you're entering an already well-served market.
And don't get me started on security/privacy issues.
> Users mostly don't care so long as the app works well enough to do what they need it to do (but they're not dumb, they'll still pick the least buggy option if there are alternatives..).
In other words: users do care. They just don't have a choice.
That's not the same than users "not minding" bad software.
>The problem that the article doesn't address
Another. Industries that are oriented towards tradeshow or holiday launches. It ain't like they're going to move NAB or Christmas just for you. Inevitable feature pruning occurs.
Having said that, I wonder how many of the great fortunes have been built on horrid, half-finished websites and associated software that were all about being first out of the gate and heavy marketing.
An interesting subcase is software that is difficult or impossible to update vs. immovable deadlines. I guess in a world that consists entirely of one silly internet surveillance marketing company after another is doesn't really matter much anymore.
October will be interesting, wonder how busy NAB will be. I know one supplier from the UK who's planning on quarantining in the carribean for 2 weeks ahead of NAB because the US still wont let people from Europe in (other than Nigel Farage)
I think that eventually the vendors will decide that it's a waste of money.
> The problem that the article doesn't address is that users don't actually seem to mind using terrible software so long as it solves the problem they face better than not using it. I could list literally hundreds of half-assed, broken, bloated applications that I've encountered in the past 25 years that have done very well simply because they kind of solve a problem a bit for the user.
This is ZERO snark, but I'm genuinely curious for lots of examples. I ask because I'm confronted with this problem ALL THE TIME working at startups that obsessively focus on visual elements that won't move the needle, versus being obsessive about solving pain better than others.
I need to start building a list that I can just pull out at a moments notice.
Would love to hear more!
> A high level of quality in software is not important unless you're entering an already well-served market.
Even then, it's not what's important. The core important thing in the product is that it does a job the customer needs done. If you write a program that provides complete feature parity with the incumbent with better engineering principles, you're going to lose because the engineering principles aren't the job the customer needs done.
Where it will make the difference is that your company will be able to respond to changing requirements better than the incumbent.
So solid software engineering is never the product for the customer. It might be the product for the company, depending if the company actually needs the improved developer efficiency and happiness it provides.
This is true, although there are other aspects. For example, if the best available solution to a problem is using software I hate then I'm going to develop a resentment towards its developers and consider them last when trying to solve future problems.
> A high level of quality in software is not important unless you're entering an already well-served market. I wish it was.
Somewhat weird mindset. It is natural that when a new market gap is discovered, the first iteration of products are crappy and do the job just barely. Applies to software, cell phones, forestry machines, water toilets. Having a mindset where everything should be perfect from the start will get you nowhere.
I agree with that last sentence, but IMO even in a well-served market things like brand loyalty, familiarity, and social momentum can often be major deterrents to switching just for the sake of quality and often times unless the major players in a market are really messing up users will not see much of a reason to switch.
A classic screed on the same topic: https://web.stanford.edu/class/cs240/old/sp2014/readings/wor...
It seems strange to me that software engineers are so frequently singled out for schedule slippage, when the impression I get is that every novel engineering project suffers from the exact same problem. Projects to design and build new military and civilian hardware and infrastructure always involve budget and schedule overruns of months or years. Can anyone provide convincing empirical evidence that software projects are delayed more than hardware projects of equivalent budget and distinctiveness from previous solutions?
A negative comparison often seems to be made between software engineering and construction, but it seems to me that the latter is a unique subfield of engineering, where you have an unusually large number of projects with a roughly homogenous set of constraints and variables. This has allowed those constraints and variables to be studied, understood and mastered to produce a discipline that more closely resembles mass-production. And in those subfields of software that also involve more homogenous constraints, such as the production of standard commercial websites, you do see a more controlled and templated approach, using tools like Wordpress.
To list just a few such slipping projects outside the software realm:
* Space Launch System
* Boston's Big Dig
* New Berlin-Brandenburg Airport
The Olkiluoto 3 nuclear power plant in Finland. It’s a new type of reactor and one of the only nuclear projects in the West.
Construction started in 2005 and the plant was supposed to be online in 2010. Eleven years later it’s still not operational.
Original budget was 3 billion euros. Current estimate is over 11 billion.
Wise moves on the part of the project proponent, had they presented their project with a 17 year construction and 11 billion euro budget it never would have been started.
Humans are optimistic. If we knew in the beginning how long something will actually take we never would start many things.
This is the "hiding hand" principle at play.
https://en.wikipedia.org/wiki/Hiding_hand_principle
I highly recommend anyone interested in "megaprojects" to listen to this EconTalk podcast episode with Bent Flyvbjerg:
https://www.econtalk.org/bent-flyvbjerg-on-megaprojects/
Very insightful conversation.
I think there's a difference between companies that build software for the market versus those that build software for a contract. The former tends to be driven by a team that believes in the product that they're trying to deliver, has a management team that has at least some vision for the software and investors that believe in the company's approach to the market. They want to make software / a product that represents the organization well and has long term value.
In the latter case, like so many government sponsored engineering projects, the company has no real long term incentive beyond what is written into the contract. Management isn't driven by the software being produced so much as meeting the contractual requirements that result in payments. In my experience, software contracting houses are a lot more likely to hold schedules over the heads of developers, and that's purely driven by the terms of the contract (billable hours). In contrast, even when I have been a contractor for a company that is building the product for itself, they tend to push for the goals of the project while ensuring it meets the needs of the company. The difference in pressure as a developer is quite substantial.
I think you'd be surprised to see how much of the roadmap at a typical "build for the market" company is driven by specific contracts. In my experience, the proportion is high even in consumer tech, and easily clears 75% for even the most commoditized of enterprise software. The stereotypical contract shop is distinguished more by poor planning and an aversion to shared architecture than any real difference in incentives.
> It seems strange to me that software engineers are so frequently singled out for schedule slippage
It follows this trend I've seen across the board in business throughout my career, of mistaking the last hair on the last yak for the entire job. The classic example is documentation. "Hey the electrical drawings are due on Thursday the Aprilteenth, can you have them ready by then?" Oh sure, that's two months away, that's plenty of time to bang out some drawings, right? But wait! The drawings depend on the functional specification. And the functional specification depends on the functional description. And the functional description is blocked on some TQs from the client. And you're still expected to produce these drawings by Thursday Aprilteenth when realistically you're not going to see even a first draft of the functional specification until the Wtf'th of August. But now it's your fault that the drawings are late because you weren't paying attention and you agreed to a due date that didn't depend on your dependencies.
I take this one particularly personally because I'm in industrial automation and, almost by definition, we get the last bite of the pie on every single project. Sure, there are some things we can start early but on the project-specific parts, generally we're waiting for everyone else to do their job before we get priority.
"Can you help us with the motion control software for AwesomeBot?"
Sure, no worries!
"Just remember we need AwesomeBot running by Thursday."
Hey cool, you present me with an AwesomeBot-shaped shell and I'll fill it with closed-loop-ey goodness.
"It's Friday, why isn't AwesomeBot doing somersaults like you told us it would?"
Uh well, AwesomeBot is lying disassembled on the mechanical workbench and half the laser cut sheet metal hasn't arrived.
"But you told us Thursday! Okay, we'll finish putting AwesomeBot together and let you know when it's done."
<fast forward 17 months>
"Hi remember us? We're AwesomeBot inc."
Oh hey how's it going with that?
"We've got AwesomeBot fully assembled and we're ready to go! Can you come down tomorrow and install the software on it so it does everything we ever dreamed of including anything we might have imagined it might do in the 17 months since we last saw you?"
Uhhhh... so how's testing going, have you switched it on yet?
"No, why would we do that? We were only waiting for you!"
I’m also in industrial automation and know the challenges of being “last” in years long projects.
We are very deliberate about setting deliverable dates in time after the required inputs by others are available, and when inevitably they are not, we have a spreadsheet sent to the customer every 2 weeks which shows what information we are missing and the delay that is causing to our progress. As we are liable for damages if we delay the project it is necessary for us to have the proof that we did not cause the project to be behind schedule. This often means having these difficult conversations only a month in to a year long project when vendor drawings we were promised are not delivered.
I "grew up" in industrial automation, I feel ya on delays. I relocated overseas for DCS project on a new chemical plant. The planned length of the assignment was 14 months...4.5 years later, we were finished. One EPC was not able to get the underground utilities installed in time before other EPCs (there were 4) started moving in pipe racks, equipment modules, etc. Imagine trying to put in all your underground cables, sewers, pipes, etc. after the site had already been built on top.
Another EPC was moving so fast to try and meet a project milestone in order to get paid, they installed an instrument enclosure totally backwards (huge crane required to drop enclosure into place). They still got paid as the milestone didn't specify the enclosure had to be installed correctly.
One of annoying aspects about controls is that non-technical stakeholders see a mechanically complete object/machine/plant and wonder why it isn't working yet. Also, I always found that the controls guys were the first to get yelled at if something at site stopped working.
> We are very deliberate about setting deliverable dates in time after the required inputs by others are available, and when inevitably they are not, we have a spreadsheet sent to the customer every 2 weeks which shows what information we are missing and the delay that is causing to our progress.
This is a great approach if you can stay that organised. It reminds me of one of my fundamental rules of business - the two benchmarks you're compared to are the previous entry in the Gantt chart and your counterpart in the client organisation. If you can stay ahead of both of those, you're golden!
When I was in the VFX world we had fixed, hard deadlines. Once the release date is printed on a movie poster it's not changing. Some parts of the process were easy to estimate since they'd been done many times before, but on every film there were some experimental new things. Often they would just end up with 2 or 3 different teams working on the same problem. One with a known technique, and one or two trying something new. Then if the new stuff didn't work or was taking too long, you could always fall back to the old way. Which wouldn't look as good or achieve the effect you wanted, but :shrug:. There was always the chance that you'd work sometimes for years though and your work would get dropped which was never that fun for the artists involved. Most development teams don't have the resources to have an A and B team working on the same problem but it's a way to contain risk.
The studios use release dates as deadlines basically to contain costs. Otherwise some directors will just keep working on a movie forever (or keep going back to it like George Lucas). Some movies, especially with VFX, are never really done, they just get to the deadline and finish with what they have at that point.
"In the eyes of those who anxiously seek perfection, a work is never truly completed—a word that for them has no sense—but abandoned; and this abandonment, of the book to the fire or to the public, whether due to weariness or to a need to deliver it for publication, is a sort of accident, comparable to the letting-go of an idea that has become so tiring or annoying that one has lost all interest in it."
–Rosalie Maggio
Is this why people have trouble with the last 5% of finishing their personal projects?
Alternate view point - driving projects to a specific deadline allows the other departments in the company to coordinate marketing, packaging, and selling the product. These people don't sit on their hands patiently waiting for engineers to finish building, their work can take many months just as the development does, and sometimes also involves making tradeoffs to deliver on time. No company wants to wait another year after development is finished before they start earning money for it.
Yeah, this is an unfortunate truth that Agile has to confront; there may be hard coordination deadlines. Or, in startups, a financial "runway".
On the other hand, setting a deadline can't force something to be possible, it can only force people to work harder and more painfully towards it. I'm sure the Amazon drone delivery failure had a date target, for example. And there have been plenty of failed "big bang" IT migrations delivered by similar immovable deadlines.
If anything this is the reality of shipping software that "agile" is designed to handle.
It's so you can (hopefully) have an educated understanding of setting that deadline, and then how you're progressing towards it. It's so if you're not on track to deliver, you can make more educated decisions about cutting scope, or adding resources (yes yes yes, i know).
Agile is fine with fixed due dates. Fixed features on the other hand...
Exactly this. You create a minimum set of features that must be shipped by the due date. You negotiate on this, adjusting the minimum set or the due date correspondingly.
The desired features beyond the minimum set are accepted as being at risk of remaining unimplemented if the due date can't be shifted further to the right. They become stretch goals that may be achieved by the deadline, or will be worked into a second effort during the maintenance of the system.
Agile (especially some of the stuff in the Lean Software world) is very well-suited to this kind of development. As are the classical evolutionary or iterative & incremental models.
You have to try to understand that building software is not like baking a cake in a custom shape. Which if you have seen those shows it requires a certain amount of ingenuity to create a cake shaped like a fountain. Or whatever.
But fundamentally software development has 1000 to 1000000 more moving pieces than a cake, generally a research project, like building a new type of aircraft engine. Except, imagine it was a really new type of engine that was being built based on an alien landing. But they could not get into the engine to see how it worked because it was an unknown alloy. So they were trying to invent a new type of anti-gravity drive based on a few clues.
I mean we are not trying to defy gravity, but there a shitton of unknown unsolved problems. You can't promise or create marketing for a feature unless we already know it's possible. For starters.
But the issue is worse than that. What generally happens is once one part of the new engine is barely working, they demand another new invention that has to be completes in the same deadline before the other one has corner cases examined even.
Sorry only just saw this.
I'm a software developer, have been for my whole career, so I know what building software is like. And while I agree that sometimes there are significant unknowns to resolve, and these can require large amounts of time, still the company has to market and prepare to sell the product before it's finished, otherwise as I said the company has to wait months or even years to get paying customers, by which time a) they'll have burned a huge amount of money without seeing any return and b) they may have missed their market opportunity. In the case of B2B transactions, the customer may have made significant operational decisions based on the deadline they've agreed to, and the supplier missing that deadline can have serious repercussions for a relationship that may have taken many years to build.
And come on, let's be honest - most of the time we're not designing a new type of aircraft engine. We're building something that looks very much like existing engines in general form, but with specifics that are different. I know it's hard, and I know estimates are a significant source of pain, but we need to have some sympathy for the rest of the company that's ultimately bringing in the piles of money that we get paid.
And you don’t really know if anyone wants an engine, although the word got banded about in some initial conversations. You could ask the customer but they want faster horses, and you are not allowed to talk to them anyway as you are assumed to not have stakeholder skills plus you are paid too much to not be coding.
Furthermore, financial penalties on late delivery are not uncommon in enterprise business-to-business settings. These penalties are sometimes quite steep
If the choice is between releasing kinda-working, but unpolished software, or throwing a developer's monthly labor cost out of the window every single day, these management anti-patterns suddenly make much more sense
I business agreement isn't an arbitrary date. Those dates should be clearly communicated to engineers.
They are communicated to engineers - in the form of deadlines...
Problem with a lot of deadlines is that you aren’t told if this is a hard requirement because of a contract or some other external factor or just an ego trip or wishful thinking of upper management.
I have seen it plenty of times that management pushed for an arbitrary deadline. The deadline passed, project was not done and nothing bad happened other than another new deadline.
There is nothing wrong with working to dead lines estimating and planing development and releasing software thats good enough not perfect.
The problem is when people start trying to change the plan half way through execution, non tech people will constantly try to do this because they are clueless its the project leads job to tell them no.
When your manager asks you "if anything can be done to pull those dates in." you say "no" problem solved, until your line manager goes to the big boss who asks the same thing and do you know what your answer is? Thats right its "no". When the big boss says you have to do something then tell him "I cant magic more time you need to cut features", big boss and manager will argue their point but unless you can do magic your point remains the same.
I know its scary being blunt with your big boss but as a senior developer thats your job.
The problem is that nothing is ever so clear cut and dry and it's always _possible_ to cut corners to rush something out, but rarely is it the right thing to do. Everyone should brush their teeth and look both ways before crossing the street, but when being chased by a killer it might make sense to skip these things. However that should be extremely rare.
The challenge is that there are no hard rules and it's highly dependent on the context. That requires mutual understanding and trust between the business and development.
> I know its scary being blunt with your big boss but as a senior developer thats your job.
This. They could hire any random chump to say they're right. If you're the expert, they hired you to tell them whether they're right.
Of course, not all managers understand this but the sooner you realise you're working for one who doesn't, the better.
Ah, you are one of those "difficult developers" :) Be blunt with your boss but also dont forget to quit if they dont change their behaviors.
Just "finishing up" a project of "6 months" where we went from reasonable to unreasonable scope, and I was the only one that said it was unlikely we were going to finish on time, weird that I was the only non-contractor....
Almost a year a shoddy version that's just copying data from their existing product is only mostly a buggy mess!
The issue is that many people are in a situation where the manager will either decide to just not listen and promise those dates or features anyway or they will be demoted,.fired or otherwise penalized for not going along with it.
That's a problem because many people do not have the luxury of quitting for another job easily.
Well, sometimes you can reduce scope to meet a deadline. It's often not advisable, but it's also sometimes necessary.
The magical scope creep fairy wants to talk to you. :)
There is a counter intuitive thing about software project estimation that took a long time for me to discover.
The more rigorously you try to analyse the problem, cutting it into smaller and smaller parts, the more error you introduce, reducing the value of the estimate at each step.
Thus, the best practice is to give a very rough estimate based on the scale of the project and your past experiences.
If you don't have previous experience with similar projects, then you should not even try to estimate.
It’s funny you say this because every software project estimate I’ve ever done has basically been to invent lots of small bricks and fiddle the numbers of these so the total matches what seems to be the intuitive size
yup, me too...then double it
Theoretically, this should be the exact opposite of the truth. Breaking a big prediction into a whole lot of smaller predictions is the basic intuition behind Fermi estimation and wisdom of crowds. As long as errors are symmetrically distributed and independent of each error, they'll cancel each other in aggregate.
The problem is estimation errors are not symmetrically distributed because engineers chronically underestimate how long something will take, and the problem is exacerbated by management pressure giving them an incentive to estimate even lower.
It’s the breaking down that is challenging, because the breaking down is the work. The correct breakdown is known when the project is completed. Until then, there are only approximate and probably wrong breakdowns.
Ever had a task with 4 seemingly “easy” parts where 3 are easy and it turns out 4 requires a big rewrite because of some hacked put in 8 years ago that are now deeply ingrained assumptions in the code?
Often you are asked to estimate a 10k part project!
They're also not symmetrically distributed because delays are much larger than surprise wins. A win of 50% and a delay of 50% is already non-symmetrical, because the delay will be quite a bit larger, and the real numbers are even worse. A 5x delay on a particular element would be unsurprising, but to estimate something and then have it surprisingly cut to 1/5th the time is something I've only seen a handful of times in my career. "Oh! There's a library in the code that already does exactly this!"
The distribution of delays is pathological, too. It's not normal or poisson or anything nicely amenable to analysis.
This is why this is counter intuitive.
We're trained to cut big problems in small parts to solve them.
This is more closely related to human psychology than pure logic.
Well it sounds like you are not doing Agile correctly.
Jokes aside I think accumulation of errors together the fact that time estimations are hard and Parkinson's law is the fatal flaws with Agile.
> Thus, the best practice is to give a very rough estimate based on the scale of the project and your past experiences.
This is my experience too. It is somehow easier to estimate big scopes then many small that sums up to the same scope. Also you get a better feeling of how far you have come when looking at the big scope then when digging down to the details.
My formula is usually to listen to the project idea, talk to a few people, take my first feeling how long it could take and then multiply by 5. This usually is pretty accurate.
Once you've been around in this software business for long enough you see the same articles written over and over again... there will always be articles about estimating software, yet there is no solution to this problem.
Anyhow, it doesn't matter. Software devs are the lowest cog, they don't get to extend the delivery date. All the layers above - management, marketing, sales etc need a date to work to to deliver their work, and they generate the $$$. No-one cares if the software is low quality, as long as it is delivered. Dev team can do further updates to bug fix after delivery. The end.
No one gets to choose the delivery date, other than actual reality. Anything before that is a prediction. The question is how good is your predictive model and do people understand the error margins. In my experience high ups like to promise dates (in general). But it’s a dance: everyone knows it’ll be later because “IT project”
It takes a lot of trust and maturity and skill (from both technical and business side) to have continual value delivery instead of estimates. This is possible but not the common case.
It is what we seem to call the 'unknown unknown'. Basically something popped up that was not planned for and now you need to research/fix/qa and shove that in. Even our 'known' bits are always underestimated. Because many people are hopeful and willing to 'try harder'.
Heck I personally just did this. I scoped out a story. Then realized I had underestimated it because there was at least 2 tech stacks in the thing I had not used much. So I had to basically drop what I was going to do and figure out some PoC on that. THEN I could write my code.
Yeah, if I could go back in time I'd tell myself "Software has always been late, it's always going to be late, you can work yourself extra-hard to make it less late, but it's not YOUR PROBLEM. You can compensate for the malfunction of the organization but you won't be thanked, you won't learn as much by hacking, and it almost never matters anyways because the startup either will get traction or it won't, it never fails by a margin of 5%"
> Once you've been around in this software business for long enough you see the same articles written over and over again
My favourite is watching successive generations discover thin client vs. fat client over and over again.
I'm not convinced an arbitrary date is a bad idea, and in fact, I continue to think it's a good one. If you don't have a point in time you are optimizing for as a developer, then why wouldn't you continue to gold plate and improve your code? Once you have a ship date, you know when it is time to knock it off and start heading downhill.
The real problem here is the leadership deficit. The manager got pushback on the estimates, and instead of explaining the reasoning and bargaining on scope, he caved and kicked the can down the road by letting the estimates crumble. Yes, estimating is hard, and you are probably going to be wrong, but once you have your finger on the scale to get a desired result, you can't blame that on the difficulty of estimating. Just deplorable leadership.
> gold plate
At my current employer, managers love to use this term. We are asked not to gold plate.
And yet they’re the least qualified to decide what is gold plating.
Case in point, gold plating in my managers’ minds means: “code reviewing, refactoring, automating code quality (linting in pipeline etc), or writing any kind of test”. To the rest of us, this is far from gold plating, nice to have, or optional. Yet this is what gets cut every single time. And then we’re asked why we suck so much and “just fixing that one little button” takes a month and breaks five other things.
Sounds like they need education on several of those points. Code review, and automated tests should be table stakes, not gold plating. Linting doesn't even seem to belong in that list because why would it add to development time?
Refactoring is a bit different in my mind. I often discover better ways to write a piece of functionality while I am writing it. Sometimes it's worth essentially starting over, and sometimes it's not. That decision IMO should be heavily influenced by business deadlines. There are times to push back the deadline because of a major refactor, but they damn well better be a rare exception, and you (and your manager) damn well better be able to explain the business reasoning behind it.
Lumping poorly judged refractors, last minute pet features, and optimizations (actual gold plating) in with critical technical practices like testing and review is probably part of the problem in your organization..
By refactoring, I mean cleaning it up once it works. But in my managers mind, as soon as the code works it’s good enough. Doing anything beyond that like sensible variable names, comments, making the code easier to read and follow is gold plating. So the second your hacked together development code works, you’re supposed to merge it and move on to the next ticket.
The pipeline stuff is gold plating because it takes longer than two minutes to configure and it then blocks merges and PRs when the code doesn’t pass linting etc (which slows us down right now, never mind that it’ll save a lot of time down the line)
I’m mostly just venting with like minded peers though. At that point, I’ve given up on fixing anything. I’ve tried, it burnt me out, and had no effect. Now I just put my 8h in, get my paycheck, and watch as we’re setting piles of money on fire. You can lead a horse to water etc.
You are free to approach this as you wish. Know that others have been there before, but most of the industry has moved on from where your organization is now. It wasn't done by digging in heels and expressing frustration, and it certainly wasn't done by giving up.
The truth is, your manager's goals are probably more aligned with yours than either of your realize. Part of growing in your career is learning how to make the connection for other people between these unintuitive things that make development faster and the goals they are trying to achieve. I wouldn't give up on finding new ways to share that understanding. It's called working on your leadership skills.
I'll caveat that there are some cases where it doesn't make sense to do the things you described. Maybe you're working on a prototype, or maybe the cash situation of the business is truly precarious. But if you want people to buy in on change, you need to understand their goals and relate to them on that level. Good luck!
Thanks! This business is doing well, it’s just entrenched in its ways, leadership is old and averse to change. I made my peace with the situation, I’m not interested in becoming a lead or anything and I’d rather save my energy for my life outside of work. I understand this doesn’t necessarily work for everyone, and I hope your replies will be useful to others in similar situation but with more desire than me to change things. Anyway, whatever works!
That's fair. I don't recommend complaining on the internet about it because it normalizes bad behavior that used to be normal but no longer is in functional organizations. There are lots of younger developers on here who don't know any better, and while we oldsters can stick it out until retirement, their careers will be much more fulfilling if they recognize the capacity to change their environments.
>If you don't have a point in time you are optimizing for as a developer, then why wouldn't you continue to gold plate and improve your code?
Because it feels good to complete work and I would like to move on to other features. How long can you work on improving a single piece of code before getting bored?
So developers should work on features until they get bored of making them better?
That's not what I am saying. The parent post makes it sound like, if a developer doesn't have a deadline he will just work on a single feature forever. I am saying that most developers intrinsically want to complete features to move on to new stuff. If they are in a good environment, arbitary deadlines are not required.
When I pick up a feature, I know roughly how long it will take and I will try to finish it within that time. I don't need an arbitary deadline from management.
> I know roughly how long it will take
You've formed your own arbitrary deadline right there, because what I was trying to say with my admittedly brief and snarky comment is that a feature being "done" is subjective.
You may have terrific business sense, and more importantly, one that is aligned with your stakeholders. That doesn't mean any other developer does. Agreeing on a deadline in many cases of a healthy team practice.
Arbitrary - based on random choice or personal whim, rather than any reason or system (first definition from google)
I think that reasonable and technically-informed deadlines are fine. But arbitrary deadlines are not a good long term approach.
We probably agree, but it's difficult to explain everything via comments :)
Right, but these dates should be intrinsically motivational, and not induce negative stress.
The article is full with interesting nuggets of wisdom, but I found it all over the place, as it mixes several issues (all of them important) into one.
The article mentiones "high-value" projects (vs "low-value" projects), but the actual distinction made is between high-effort (new framework, new kind of product) and low-effort. Value and effort are not neccessarily correlated, indeed by the end of the article the team has had a high-effort project that resulted in a low-value product.
It also conflates product discovery (what will actually have value to the users) and technical discovery (how do we build the damn thing). It starts with the technical challenges (and I totally subscribe to the notion that estimating anything that you're not doing routinely is Dangerous, see https://blog.senko.net/bridges-vs-apps) but then veers off into describing mismatch between features built and value add (also a very important subject, but a different one).
I would love to see author expand each of the sections into an own article and explore these things in more depth!
The most underwhelming part for me are the recommendations at the conclusion. So (focusing here on tech aspects) if "Something by Date (tm)" methodology is bad (which I can agree with in principle), the solution to "let competent engineers prototype [the app] in 3 or 4 different frameworks", ie . "give your engineers unbounded time"? All developers (and I am one) wish they'd have more time to implement things properly, so that's not a new insight. But it's not actionable either - "give more time" then turns into "Something by Later Date (tm)", which is basically the same problem, just with more money (and thus stakes) involved.
I hoped to see an exploration into how to creatively solve for "can we deliver value" + "we have this budget and time" constraints.
Chapter 11. Plan to Throw One Away
Chemical engineers learned long ago that a process that works in the laboratory cannot be implemented in a factory in only one step. An intermediate step called the pilot plant is necessary to give experience in scaling quantities up and in operating in nonprotective environments. For example, a laboratory process for desalting water will be tested in a pilot plant of 10,000 gallon/day capacity before being used for a 2,000,000 gallon/day community water system.
Programming system builders have also been exposed to this lesson, but it seems to have not yet been learned. Project after project designs a set of algorithms and then plunges into construction of customer-deliverable software on a schedule that demands delivery of the first thing built.
In most projects, the first system built is barely usable. It may be too slow, too big, awkward to use, or all three. There is no alternative but to start again, smarting but smarter, and build a redesigned version in which these problems are solved. The discard and redesign may be done in one lump, or it may be done piece-by-piece. But all large-system experience shows that it will be done.[2] Where a new system concept or new technology is used, one has to build a system to throw away, for even the best planning is not so omniscient as to get it right the first time.
The management question, therefore, is not whether to build a pilot system and throw it away. You will do that. The only question is whether to plan in advance to build a throwaway, or to promise to deliver the throwaway to customers. Seen this way, the answer is much clearer. Delivering that throwaway to customers buys time, but it does so only at the cost of agony for the user, distraction for the builders while they do the redesign, and a bad reputation for the product that the best redesign will find hard to live down.
Hence plan to throw one away; you will, anyhow.
-- Frederick Brooks, The Mythical Man-Month
this is great comment!
i must say, the mythical man-month should be a must-read* for any technical managers or execs in a tech org (hmmm throw devs in that list too)
*and maybe re-read every year to keep it fresh in mind
Which is more likely: Businesses invent truly arbitrary deadlines for the hell of it, or the “engineers” don’t want to pay attention to business requirements and competitive pressures?
When so-called engineers stop spending half the development schedule choosing a framework and the other half trying to make their dev setup work on everyone’s personalized laptop they will have some credibility complaining about “arbitrary” business goals and requirements.
Do you want developers that are passionate about their work or do you want compliant drones that will only ever do the bare minimum for you and blindly follow the directions laid out by management, even when they know it will be a disaster? I see a lot of companies where management just keep pushing their employees to get a month's work done in a few days, cutting corners and creating issues because nothing is properly thought out or tested.
It's called "development" not "Bam, it's done!" The best companies allow proper time for the development process to take place and have a proper team structure. Companies where everything is discussed as a team and you succeed and fail as a team. Proper agile teams which communicate at every step, with developers, testers, a scrum master, a business owner and a manager.
If you don't have that, you're basically not a proper development team, you're more of a startup that can't afford a proper development team. Nothing wrong with startups, but if you are a startup then you have to deal with your own weaknesses and shortcomings and take your failures on the chin.
From anecdotal evidence, a) is more likely than b). The deadlines are arbitrary because most companies are slow bureaucratic oil tankers that are difficult to steer and will mow down everything in their path once a course is set. Management's job should to be to work on this, to improve the system as a whole and making it more nimble, not on myopic deadlines that fulfil some checkbox and are tied to your EOY bonus. This is outlined by people like Deming and others in the systems thinking space.
From experience, in their career, engineers not only need to excel technically, but are also forced to pick up everything from UX, methodological BS (from Scrum to Itil) to domain specific know-how in multiple fields or areas. Since many managers do not know what they're doing, senior engineers often times end up being de facto management consultants as well. If you are working in a business environment (as opposed to writing low-level drivers or whatever) it's almost impossible to not pick up on what's going on around you.
How many banking managers know how to code? How many engineers working in banking know at least something about the processes, compliance issues, how the org is structured, what the competitors are?
The stereotype of the autistic programmer who is only interested in shiny gadgets and tech needs to die.
I agree, except the programmer (autistic or not) who wants to be left alone to code and play with shiny things is not just a stereotype. Just read through HN any day to see posts about ignoring meetings, marketing, management, business priorities and focusing excessively on languages, tools, the “best” ways to do things, dismissing non-programmers as hopelessly useless (“many managers do not know what they’re doing.”)
As a freelancer I have to get to know the business my customers are in, I can’t just focus on purely technical things most of the time. Many times I have described my work to other programmers and heard how they want to be left alone to code. I take over legacy software and failed projects for a living. The two main reasons in my experience for software dev project failure are (a) developers did not bother to gather and understand requirements but rushed to start coding, and (b) poor communication with the customer and stakeholders. Those faults may come from arrogance or inexperience, or both.
> (a) developers did not bother to gather and understand requirements but rushed to start coding
Because of arbitrary and too short deadlines
> (b) poor communication with the customer and stakeholders
Communication is a two way street. Why blame devs for all this?
In my opinion, when a project fails, you have to blame the people higher up in the management chain who are coordinating the work, rather than the engineers. (in a large company)
If engineers are to blame, maybe look at your hiring standards and hire better engineers. This points back to the management again.
> Those faults may come from arrogance or inexperience, or both.
Don't hire arrogant or inexperienced engineers then?
Management may be at fault, but anecdotally I see programmers rushing to code very often. Almost every one of my customers has the same story: the last developers stopped answering emails and calls when the project ran into problems.
I have worked with terrible managers an dysfunctional organizations, but I have seen developers rush to code and stop communicating far more frequently.
I don’t really care about blame. These are people problems with no single solution. When I get involved in a failed project the customer has moved past blame and just wants to salvage something to meet their requirements. The programmers will blame management with little introspection about their own role in the failure.
I do disagree. The primary responsibility of the manager is to manage and to manage they need to understand the process. The problem is that managers very often do not understand the software engineering field particularly well at all. I've seen it a thousand times that both large and small issues are simply ignored by managers, because they either do not understand them or even if they manage to grasp something, they fail to understand that they need to actively do something about it or just plain out ignore the issue, either with wishful thinking or just a power move where they think the issue isn't theirs to solve and in fact that the issue lies on the engineer to resolve. Most engineers are highly logical and would not bring something up without good reasons. It will have a negative impact if not handled.
I do not manage like this myself and I do not blame my engineers for anything that sits on my head, but I learned these lessons painstakingly through 15+ years of professional coding myself, seeing all kinds of roles at all kinds of companies creating chaos, project managers, product managers, bad managers, bad manager's managers, etc, but also the good ones luckily, the ones that were respected, knew what they were talking about, won the hearts of engineers, etc. This is what I modelled my own path after.
> Just read through HN any day to see posts
Only an incredibly tiny slice of professional engineers are on HN, even smaller slice of that actually post anything. HN is not representative of anything.
For most of my 40 year career as a programmer HN didn’t exist, but the complaints about management and business priorities did. And so did those stereotypical developers who want to be left alone to play with shiny things.
You can find the attitude I described all over the place, not just HN.
When I get involved in a failed or failing project the developers almost always blame management, schedules, budgets, marketing priorities. They almost never reflect on the time they’ve wasted or how they don’t really understand the goals or users for the software they are responsible for.
> They almost never reflect on the time they’ve wasted or how they don’t really understand the goals or users for the software they are responsible for.
I think the quoted statement tends to be a systematic issue on struggling projects that describes the decision makers, regardless of role.
I'm not sure how anybody can expect other outcomes than a few lucky successes and a lot of failures when understanding why you are building something seems to be undervalued by management, technical or other.
So, projects fail because of (arrogant or inexperienced) developers but not because of managers. And then you wonder why engineers say “many managers do not know what they are doing”.
I think every case is individual. One of my recent gigs had management failures and engineering failures.
Management failed to communicate the limitations of the investment by investors to finish a project. In this case 4 months window of having an additional small team to build it. I had to pay very close attention to extract that information, my inquiries into it were rebuffed. My contract held the most clues.
Engineers didn't bother to listen to the clues and started a design by committee loop that was going to take a year+ to finish. They were much more interested in new shiney things than the financial business restrictions and this attitude was enabled by management.
We managed to finish something within the investment period only because I pushed and prodded at every turn trying to finalize decisions and cut out scope, without managements help, to do so. The manager who tried to shield engineering from deadlines didn't help this despite the train wreck I tried to communicate. At the end of the investment period all the extra engineering hired for the task were let go and the regulars were left to finish the demo we managed while maintaining their normal work load. It is now almost a year later it still isn't released.
While there are places where tone deaf management creates death matches for no good reason. There are apparently also places where management let's engineering walk all over them like a spoiled child with no grounding in financial realities.
> The stereotype of the autistic programmer who is only interested in shiny gadgets and tech needs to die.
The stereotype has been kicked to death, dismembered, burned, rebuilt as an effigy and burned again throughout the years.
The stereotype of the evil and incompetent manager who is only interested in ruining the lives of their underlings, who would solve the world's problems if only left alone to do their jobs, is still going strong.
I agree with you here, but some observations:
Managers are imbued with more formal power than their reports and as such should be held to a higher standard. Simply passing on pressure and not standing up for your team will never inspire loyalty.
Taylorism and top-down management are culturally embedded in many companies. With software eating the world you have companies who are ill-equipped to deal with other ways of working and do not understand how writing software is fundamentally different than building a house. This all inevitably leads to cultural conflicts.
I often see engineers having a cynical self defeating view of the business. They will blindly follow complex requirements while complaining that there is not much they can do about it.
But if you demonstrate that you understand things from the business perspective, they are much more receptive to making adjustments.
Product managers like to gold plate things as much as engineers. There is usually some fat you can trim from the scope to make a deadline.
Businesses—in the guise of mediocre managers—absolutely invent arbitrary deadlines as a mechanism of control. If the deadline were real, it would have context, engineers could use that context to adjust the scope and design of what they're building and we wouldn't be having these conversations in the first place.
The problem begins when deadlines are used to limit agency rather than support it.
Well the counter argument is that shipping later also destroys value (because value can only actually start to be accumulated once a product has shipped).
But this article doesn't seem to address the obvious counter-point.
For instance, take the below:
> wouldn't you rather run a your customer's payments through code that is at least attempting to handle error conditions, rather than some happy path code that just assumes everything works?
But this doesn't address the obvious issue which is, there is value destroyed (or not created) by not allowing customers to pay you earlier.
I really don't like this conclusion. I guess this is why we need financial regulation, because rushing through a buggy payment system so you can make a buck quicker is how customers get their accounts hijacked and identities stolen. Maybe as an early-stage investor or company founder, it makes no difference to you because you'll exit before anything ever hits the fan and the law will never hold you accountable, but you might be ruining people's lives. Certain systems need to be robust even if it means businesses might have to earn revenue more slowly. You're not creating value in this case. You're stealing it from your customers.
The article is talking about integrating a payment gateway, so I don't think this specific risk is valid but the overall point is right - yes, there is a trade off between speed to market and how many workarounds will be required.
The truth is that sometimes speed to market is more important than quality, and visa-versa. Handling sensitive credit card data? Sure quality and security are important of course. Implementing a payment gateway to bootstrap a company where that is 1 of 100 tasks before go-live? Hack away and go as fast as possible, and worry about improving it later!
"destroyed" and "not created" are not synonyms, they can't be used interchangeably as you do here. This invalidates the argument for me to be honest.
Well the article talks about examples of 'destroying value' being projects that failed to go live, which would be 'value not created' to your point (it's not my choice of language, I was just following the article!). If it invalidates my argument I assume it invalidates the article too.
"Our boss can't judge the quality of our work, but he knows when it's late" -- Dilbert
I don't think even the great Yogi Berra could have said it better.
The whole point of organizing things in the first place, is to reduce or remove uncertainty and risk, which cannot be done with software projects.
Software engineering is similar to writing a book, or a movie script; but it is mistaken for some advanced reorganization of numbers in an Excel sheet.
Normal leadership training relies heavily on deadlines to "move things forward", or force us to "see where we are" etc. This only creates stress on the creators themselves, making them far worse at actually creating something. This is something all developers know, and this is why companies started by developers often have a very relaxed attitude towards work. I.e. the dart board, pool table, or beer taps are not there to make the office "look cool", it's there to make a relaxed atmosphere, where you work when you are "in the zone", and relax when you are not.
Everyone who writes movie scripts, books or software knows that you can do more in 4 hours of when you are in "the zone", than in 80 hours in an open landscape, where days are split into short work hours, disturbed by status meetings about progress etc.
Work expands to fill the time you have. If you don't set a goal, you'll never ship. Estimation is not an exact science, but, I rather set a goal post that I can move if I need to than not have one at all.
"Parkinson's law is the adage that "work expands so as to fill the time available for its completion". It is sometimes applied to the growth of bureaucracy in an organization."
The problem is that negotiations phase where the devs get screwed down on the “price”. This is of course a silly thing for management to do given they are paying by the hour anyway!
There are three kinds of deadlines: fake deadlines, contract/revenue driven deadlines and reality driven deadlines. Most problems with deadlines I've seen come from chronically mislabeling the deadline (e.g. sales don't close, so engineers start ignoring revenue driven deadlines). Fake deadlines sometimes are necessary to get things to some finished state, but they should be used very sparingly. Deadlines should be honest, and when something changes (e.g. some other team is going to be 3 weeks late on their part of the product so being on time doesn't matter) the deadlines should be quickly adjusted. Deadlines are a work-planning and asset allocation constraint, and will cause very bad decisions if they are not understood and managed well.
A better solution ten years later might not be a solution at all when running a business. So it cannot be just open ended.
The point I take away from this more: poor planning and execution processes destroy value. It is not an "us" vs "them". Have clear responsibilities for removing internal/political blockages, for allowing testing, etc. How to manage new delays and unexpected issues? Who can decide on changes? Organizations can manage to hit target with a reasonable product/outcome.
this seems to mainly be relating to pushing to an arbitrary date for creating something new but still sometimes dates are given to us because of some sort of regulatory reason or a company contract meaning that something needs to be done by a certain time - before anyone says don't make contracts like that: The Danish and Swedish parts of Thomson Reuters WestLaw were sold off to an English holding company, as part of the sale the sold off parts were allowed to remain on WestLaw for one year after which they would have had to pay approx. $165,000 dollars a month.
Obviously we got off of by that arbitrary date, and if we hadn't that would have been a value destroying mistake.
on edit: the weird monetary amount is my attempt to turn to U.S.D the DKK amount at the time which was 1 million dkk - iirc the dollar was at a low but it was between 161 - 165.
Also government contracts, at least in the US, have to be authorized by an appropriations bill from Congress. A federal agency can't just go to Congress and ask for an unknown amount of money to deliver a capability at an unknown date. Whether it ends up being right or not, and probably it usually isn't, appropriations bills have to be time-bounded and include a maximum dollar amount, and the awarded contract can't go longer or higher no matter what the engineers think it will actually take.
In which case it would be wise to be extremely conservative in your estimation, taking initial engineering estimates and padding them further, not whittling them down. Because a baby takes nine months and can't be safely rushed out in six no matter the acts of congress, presidential edicts, and demands from the grand poobah.
If you fix the dates, something else has to give. If time, cost, and scope are all fixed you're going to fail, almost guaranteed. There are always unknowns and in this scenario the only thing you've allowed to flex is quality.
I'm not saying you can't come in on time and under budget. This article spells out the dysfunctional dynamic of date pressure. By creating this much pressure from the beginning, you create an environment when corners are cut from the beginning and the problem compounds over time. The pressure-cooker mindset of management kills the quality of the project, even if the estimates did turn out to be reasonable.
> I'm not saying you can't come in on time and under budget.
Isn’t this what we tell our managers to make them feel better about our estimates? I’ve never ever seen or heard of a software project being delivered on time and under budget.
To be fair, Engineers engineer and wouldn't mind spending twenty years engineering and discovering new paradigms and interesting things to get 1.2% gains. Deadlines force the issue to put the wandering mind of the engineer in a straight-ish path. Not perfect, but the alternative would take a lifetime.
There is never going to be one clear way to deal with scheduling around software projects. The number of variables involved is vast.
I view schedules around software projects as a few different things:
- Public expectations for specific deliverables that your customers can align to, many times with a calendar unique to each customer.
- Internal carrot/stick for managing constraints around active feature development. If you have an infinity budget (film/AAA gaming), you might be able to run less-bounded parallel efforts as noted elsewhere in this thread.
- Orthogonal operational concerns (planned outages, et. al.)
- Roadmap for strategic product development that the investors can think about
So, when someone says something to me like "are we still on track for end of the week?", I have to provide an extremely qualified series of responses and ask more questions.
I always tell my PM:
You can pick a feature set or a launch date. You can't pick both.
My team tried that too. We were laughed out of the room. The project failed. We’re starting a new project with exactly the same mindset. But this time it’s different, I’m sure.
I’ve lived through a project managed like this. The problem isn’t the date. It’s the unwillingness to prioritize and cut scope when the bottoms up estimates show your first version is too big.
> The Big Boss is a pressure kind of guy, he believes that people work best when they are under pressure and if pressure isn't applied, they waste a lot of time on unnecessary things. [...] he goes away thinking what he already knew, that engineering is just screwing around instead of getting the project done.
Sadly, he's often not wrong. My own work ethic isn't exemplary, and sometimes in the absence of pressure I slack off. I'm surely not the only one. I don't know what to do about it though.
Reframe the question: is it slacking off or is it restorative mental break?
Just speaking for myself, but I think better when I'm not pinched between two projects both demanding preference and priority over the other. I can maintain a clearer mind when I don't have people breathing down my neck about deadlines they set and then broke.
Sure, one learns to navigate those situations and maintain an even keel, but still one is obligated to respond in a professional manner to those inquiries and meetings and other emotionally manipulative games people play to get their way when in a crunch. Working through those things still requires time out of day and energy out of mind. Both of those latter resources are limited.
This is a combination of management not asking for input, pulling a deadline out of their asses, and engineers chronically low-balling their own estimates (while the managers think it's the opposite).
My pet peeve is when a deadline is a "round" date. 1st of the month, 1st of the year. WHY? Your users don't give a shit. If you are not publicly committed to a launch date, which you should never be, until your product is ready.
That said, internal deadlines are a moronic thing to have. These should be goals.
Yep. I’ve preached this whenever I talk about dev process. The best way I’ve heard it framed was using two different terms: deadline and target date.
Target dates are internal goals, not tied to anything other than a rough estimate or a day somebody made up.
Deadlines are tied to contractual obligations, events, etc.
If it’s not a real deadline it’s a target and targets can move. Real deadlines mean you start talking about how to trim your requirements to have a shot at hitting it.
In my experience "target dates" have a funny way of transforming themselves into deadlines over time.
It's the confusion of estimates and plans with commitments. "How long will this take?" "About a week" "Great, ship it Monday morning" "Wait, if it has to be Monday then let's spend 30 minutes figuring out what needs to be shipped and what can slip until a later update release" "No, you said a week".
Like most things, it depends on leadership beating the dead horse to constantly remind people of the difference.
The most effective manager-engineer relationships I've seen are built upon a problem-solving culture, rather than feature-shipping. If your goal is to solve problem X in Q3, your managers and engineers have far more flexibility to collaborate and compromise. If your goal is to ship X features in Q3, your managers have less flexibility and your engineers have an incentive to cut corners.
time, scope, quality, cost. pick 3, the 4th will be a function of the 3 parameters you picked. So you need to understand what's important.
If you're building a Mars lander, time is everything because if you miss your launch window, you need to wait 18 months for the next one. Most of us are doing something else.
Author is wrong about the underlying cause of that disaster.
From my point of view, the cause is different:
1. Lack of design a.k.a. designed by coders > At 5 months the team has happy path coded the entire feature list, the screens are ugly and non-intuitive, ...
It's obvious, that the product wasn't designed. Feature list isn't design. Competitors app isn't design. When you have UX/UI/API designed, it doesn't happen that the result product is ugly because of the lack of time. Lack of time may cause, that there will be nicely looking, but slow/incomplete/unreliable product. You need blueprints before building. It can be designed upfront by one designer who is in contact with user and stakeholders. When not, it has to be designed during coding by software engineers which are drowning in technical details and cut off from users. So just having some more time wouldn't make it a good product.
Also with some plan, they could have created estimation based on something. That would help them to get a realistic timeline.
In the end, the boss and PM should have known better.
Given that people move jobs quickly anyway, it is not clear that value matters. Rather, I need the appearance of value by X and if that is done by mortgaging the future, well, that is the next guy's problem.
That's a problem, not a solution.
I agree that it is a problem. I just think the root cause here is driven heavily by how careers increasingly work.
Deadlines are a fact of life. Without them, nothing would get done in a timely fashion. If you do not know how to work to a deadline, you are not a professional.
The diagrams in this article are a value destroying mistake
I know it's just an illustration, but that "Code done with care" section is making me twitch.
No transaction context to rollback if one of the steps fails.
It’s like skeumorphism. It’s not ideal, but it’s a common or goal reference point that doesn’t need exhaustive explanation.
Can someone tell the engineers I'll be happy to give them unlimited time if they will give me unlimited value?
Given unlimited time you can have unlimited anything, no?
Just-in-time software has been a thing for a long time. It's not a mistake.
Love the format of the talk converted to the text