Symptoms of Dysfunction in Software Teams
blog.hedges.netIt was a neat prototype, you know. A few loose ends here and there, but a solid proof of concept. Executives loved it. Could see the $_$ in their eyes. So they had us plug the holes and get it to production as fast as possible. Time to market, that sort of things. I recall John didn't like it. Caused quite a stir. But who was he to decide anyway?
[…]
Yeah, that became quite the thing. Lots of new features. Funny thing is, they used to come fast. Not so much lately. Did ask why —were simple features—, but all I got was some bullshit about about "technical debt". Don't give a crap is what they do. No matter, we're entering cash cow mode now. Champaign?
[…]
I'd like to, but John left (something about a "big ball of mud"), and Bob got promoted… That leaves Jane and Sally. Still many glitches to fix, but they will do —not like I have a choice. Besides, they know the billing subsystem inside out, and women are good at cleaning things up, right? Wait, I didn't —are you recording this?
[…]
Phew, that was a close call. Jane warned IT about that OS update, but they kept talking about this "security update". Four days without revenue! Jane and Sally were no good, even Bob couldn't see what was wrong. Had to call John back, you know. Fixed the problem in a pinch. I hear HR made an offer, but I guess there was no changing his mind. Oh well, at least things are back to normal.
[…]
Dunno why, but this systems scares me a little bit.
I was on a project that produced a carcinogenic prototype. The irony was we knew it at the time. Before it was shown to the higher-ups, us developers joked that they'd like the prototype so much they'd come back with a completely unrealistic deadline for the finished product.
They did. They wanted it ready in 3 months. I estimated 10.
We delivered something - I'd hesitate to call it a finished product - 15 months later, and it was deservedly panned by users.
I've spent a lot of time since pondering exactly where it all started to go wrong, and I'm entirely convinced the day we demonstrated the prototype was it.
I saw a startup do a tablet once (a very early one). They'd written a lot of demo code in Smalltalk V.
I interviewed there, and turned down an offer. When I was talking with them, I joked that they needed to be careful or they'd wind up shipping their demo code. "Oh, don't worry," they said, "We're going to re-write all of that."
A year later they shipped their demo code. A year after that they'd sunk without a trace.
(I sometimes wonder if they would have had a chance if I'd joined them. I'm guessing it would have turned out pretty much the same).
"I sometimes wonder if they would have had a chance if I'd joined them."
Maybe if you joined as VP of Engineering or CTO. As an IC, all you're going to do is get dragged down with them, and go home pissed off every night.
I've been there. A company has no existing test infrastructure, it's all done by hand, and the build "system" are the binaries that come off a dev's machine? No problem! I'm you're man, I'll have you up and automated in a jiffy. Except that the reason there's no existing infrastructure now is because of general cluelessness, not for lack of expertise.
You need someone to hook the hoozit that I lightly touched last time I was there to your new whatzit? Yeah, I'll come in on a contract basis to get it sorted. My first clue should be that none of the other devs who know it better than I came to your rescue. No specs for the thing I'm supposed to integrate with, no documentation as to what it is you actually want, and a PM that sits and watches YouTube videos all day? Yeah, I should have guessed.
It's top-down, and if you're not at the top, you're not going to save them from themselves.
Not at all - the day you went wrong was the day you wrote code that worked and assumed that the people paying you would let you rewrite it later.
That never happens, unless you force the issue by making the first version fundamentally unsalvageable through choice of language or something like that.
If you don't know for sure that you can't (for a very real reason, like the fact that AS3 won't run on iOS) ship a line of code, it's on you to assume that everything you write is going to get to production. That's the reality, even if it's uncomfortable.
...I say as the lead on a game that just got sent in for review today, 8 months early, with 95% of the "prototype" code that we started a few months ago intact. Luckily we never believed that it was actually prototype code, because wasting a month and a half to recreate something that already works is a hard sell to anyone that hasn't struggled with the codebase already.
Pad estimates, and plan for the worst, is all I can suggest. But also realize that sometimes short timelines are a blessing: the people that would be filling up the feature request list stop doing so once you tell them that you're already overcommitted by 80%. That helps the product, usually.
We were asked to produce a prototype in a two month time frame and we did. It was the subsequent unrealistic time frame after we demonstrated the prototype that did the damage.
The 'it depends' answers are good and I would not cite them as evidence of a problem. You employee people who understand what they are talking about and who also understand the importance of an accurate answer. They are probably very good engineers.
How does X close a TCP connection... it depends on the operating system in question. What cipher does my browser use when talking to X website... it depends on what ciphers are supported/available and how the client/server are configured. Which router do these packets go through... it depends. Is my password secure... it depends on the string you chose, the hash type used to store the password and who the attacker is and what their resources and time frames are.
There is hardly anything absolute in technology/software. And, people who want an absolute answer are only indicating that they do not understand the fundamental complexity issues that we deal with as technologists.
That didn't seem his point. He didn't say at all that the person providing the answer was not competent or being imprecise or whatever negative. The example he gave was different that yours: it was about the software the team is responsible for and about the fact that technical debt has been formed. In dysfunctional organizations, dysfunctional software happens despite of the bright people.
"In dysfunctional organizations, dysfunctional software happens despite of the bright people."
A variation of Conway's law. https://en.wikipedia.org/wiki/Conways_Law
The potential corollary: "In functional organizations, functional software happens even when the people aren't that bright."
Exciting or sad, or both?
>"In functional organizations, functional software happens even when the people aren't that bright."
It could happen, but I've never seen it, nor have I even read about it the literature. The sum conclusion of 20 years of software engineering literature is that two things matter: the quality of your programmers and the stability of your requirements. Of those two, the first matters more than the second. Get those two, and it doesn't matter what methodology you use, you'll have functional and elegant software. Miss those two, and it doesn't matter what methodology you use, your software will be crap.
> the quality of your programmers and the stability of your requirements. ... Get those two, and it doesn't matter what methodology you use, you'll have functional and elegant software.
I've always believed that when excellent programmers advocate whatever methodology they use (e.g. Beck et al.) they themselves are missing the fundamental fact that they could produce excellent software using three sea shells.
I'm really glad that those excellent and public programmers don't have a more perverse sense of humor.
... or, maybe they do ...
The 'it depends' answers are good and I would not cite them as evidence of a problem.
They might be the best answers you could get in that company, but it would be better still to get:
"Use this library feature"
"Ok; are there any weird exceptions?"
"No, this is it. We rolled all that company's accounts into this system when we merged, all old accounts in when we upgraded, and planned and designed this to be good enough for the high risk accounts too. There's only a handful of other accounts kept separately and there is no way to verify those from here, by design."
I once worked with a guy who gave a lot of "it depends" answers. (In fairness, he was the chief scientist, and there was a lot more of valid "it depends" than I wanted to recognize in the way microwaves interact with living tissue.)
One day a coworker and I asked him a question, and demanded that he give us a yes or no answer. He thought for a moment, and then replied: "Yes. However..."
[Edit: Fixed typo.]
I've been stuck maintaining a carcinogenic prototype for the last 2 months. While I don't like dealing with the maintenance nightmare, I do really like that term. I'd never heard it before, but I knew what it was as soon as I read it -- the text in that section only confirmed my suspicion.
Yeah, this definitely feels like a permanent addition to the lexicon. It crisply describes at least two systems I'm maintaining currently.
If it gets canonized, I'd much prefer "cancerous prototype" because it's easier to say.
Also known as: Quick Hack, Kleenex Code, Disposable Code, Scripting, Killer Demo, Permanent Prototype, Boomtown. [1]
http://blog.codinghorror.com/the-big-ball-of-mud-and-other-a...
"There is nothing more permanent than a temporary solution".
"metastatic prototype"
You mean metatastic prototype?
Hmm, disagree. The prototype doesn't have cancer, it causes the cancer. You're right that it's easier to say but it's saying the wrong thing.
I like this better too - the prototype will inevitably grow from its humble origins, and with every new feature that is hastily thrown in, the application becomes ever so much riskier.
Prototumor
The easiest heuristic is "Are they changing ticketing systems twice per year?"
I've never seen a competent team spend any amount of time chasing after ticketing system nirvana.
Agree. The last company I worked for used salesforce, email and excel to manage a 1 million line enterprise behemoth. Without much hesitation as I was employed to fix the company, with zero budget we fired up trac and integrated it with AD and their reporting stuff. Now this was frowned upon (open source!?!?! Never!) but it delivered real results and traceability. Now a year down the line and 5000 tickets later they finally accepted the value and assigned a PM to look at the company processes which were clearly broken.
A month later a totally broken copy of JIRA appeared whilst I was on holiday with the old process that was broken crudely crammed into it with no reporting, no backup/restore and no amount of crazy spared. Yep clearly the problem was the issue tracker not the process.
Now $32000 a year in protection money from Atlassian hits the upper management team (as well as the process brick wall which bottlenecks everything on the PM) so they decided that the problem is:
"We need another ticketing system. Sales force is looking good if we can just customise it for our process."
So I quit with much "fuuuuuuuu..."
Sounds about right.
I briefly worked at a place migrating from Trac to Jira. Oh, and they still had some stuff in Bugzilla. This team wasn't more than 2 years old.
I disagree a bit with these points. Sure, ideally you'd avoid all of these issues and having these issues several years into a product is certainly bad. But when a new product is just getting off the ground, there is a hefty cost to Doing It Right. I think many successful products would never have launched if a scary number of corners weren't cut to get version 1.0 out the door.
Depends what kind of product. If you're a start-up launching for Christmas, sure. But if it's a contract to deliver some long term custom product at a multi-year deadline (I have worked on such things), then you should definitely not push that prototype.
(Actually, the inverse error is often made: trying to get it right the first time around, which is impossible. A prototype is even worse, but least you can test your major ideas, so you can see which are good, and which are hopeless.)
I think the "prototype" analogue borrowed from engineering to software is a bit problematic. I don't believe there is such a thing as a "working prototype" in software. It either works or it doesn't. If it works and that's what business needs in the production environment, then ship it already. Nothing wrong with that.
The problems start arising when the issues of scalability and maintenance were not considered part of the prototype development. And this problem does not only affect "carcinogenic prototypes" [sic] ... such problems manifest, and do so regularly, with software projects that had multi-year deadlines and good intentions all-round vis-à-vis scalability and easy maintenance.
These aren't technical issues. These are issues of process and management.
I strongly doubt they have much unique to software development systems, to be honest.
Yeah... that post hit really hard. I got to quit my job or I will go crazy.
Dependencies - Check
Jane Doe - Check
Town Hero - Hey look! It's me!
Carcinogenic Prototype - Yeah sure, that's the one Jane Doe is maintaining!
The hero is negligent in their duties and intentionally leave an app is a poor state to promote their importance. Are you sure you are the hero?
The hero is called in to patch up some flaming wreckage. Mgmt never allocates the resources to actually fix the root cause. Thus the hero is seldom at fault, otherwise they would be Jane Doe. There is a specific type of hero who deliberately creates this situation, but IME they are not the rule.
Source: I was That Guy for about 8 years at three different gigs. Now I specifically negotiate no afterhours support in my job interviews, and turn off my phone when on holiday.
yes I am. Me and all my coworkers, are actually quite good but are doing a shitty job. I'm included.
The shitty part is that they pay me above the market.
I am new to the industry and have only really had limited experience. Is it more fair to call these symptoms of dysfunction or business as usual. How do you begin to avoid these problems?
In my experience, one easy way to determine how good the job is going to be is based on the pecking order of various departments in the company. At a company that does software, the software developers are usually at or near the top. Alternatively, at a marketing company, the marketers are at the top and the software developers may very well be near the bottom.
You can gather this information in a number of ways in an interview. Simply ask one of the interviewers where their group exists in the company hierarchy. If you don't want to be explicit, ask to look at the area where the developers sit and work. If the area is well done and people seem happy, you're probably in good shape. If the people are essentially stuffed into a closet or makeshift office/hallway then you might want to run.
Generally, try to work for a company that makes money on software or technology. A company that makes money on other products or services is likely to treat software developers as a low-level function of the organization.
> Generally, try to work for a company that makes money on software or technology. A company that makes money on other products or services is likely to treat software developers as a low-level function of the organization.
I'd say this is usually good advice, but with a caveat: larger companies with deep chains of command -- even software companies -- typically have "management culture". These types of companies put a lot of pressure on their employees to move into management, even if they're not good at it or they don't want to. These companies reward playing social power games above any other kind of achievement. There will be lots of meetings and not a lot of work being done. There will be no reward for technological innovation or productivity.
So, I'd say avoid companies with deep management hierarchies unless 1) you want to go into management or 2) are stoked by the thought of doing the bare minimum.
The reason I asked is effectively because your comment and the OP more or less describe the type of company I work for.
Some people thrive in such an environment. For those people, it's certainly not dysfunctional. Many companies with such deep hierarchies are in no immediate danger of going under -- they got so massive by being successful, after all. I think that once a company reaches a particular size, they stop trying to succeed and start trying not to fail. Again, some people really do enjoy that environment.
To me, it felt like a psychological prison. I managed to break out.
The only advice I can give is this: if you're unhappy where you are, start planning your escape. Start networking and be patient; in my experience, this is the way to land a good job. On the other hand, if you're content at your current company then who cares what other people call dysfunctional?
"On the other hand, if you're content at your current company then who cares what other people call dysfunctional?"
That can induce bad habits.
People look askance at resumes from successful Microsoft employees because to succeed in the last N years they had to do things that add no value to anything other than surviving or thriving at that dysfunctional company. Why hire such an person if there's a good chance he'll do some damage to your company while you attempt to deprogram him, during which he'll be much less productive that someone from a less damaged company.
(OK, finding the latter might actually be difficult....)
You're absolutely correct, but I'll answer your question:
> Why hire such an person if there's a good chance he'll do some damage to your company while you attempt to deprogram him, during which he'll be much less productive that someone from a less damaged company.
Hire such a person if you happen to be hiring for a dysfunctional company. They've got the experience, and they'll appear productive immediately.
So then the "black mark" on the resume only becomes a hindrance if after, say, 10 years a person finally decides to break out. It's still possible, but it will take a lot more work.
This isn't necessarily true. I'm the one-man maintainer of the makeup company I work for's Rails site. It's a pretty sweet gig. The marketing team's turnaround doesn't even come close to mine, so I get a lot of downtime from adding new features. They literally can't think of things to implement faster than it takes for me to do them.
There's a lot of technical debt, but I have the time to clean it up. Most of the previous guys in my job just coasted, so I'm already kicking ass compared to them.
That's the thing: technical debt is the cost of innovation.
Edit: "Technical debt has to be the acceptable cost of rapid and affordable innovation."
We can't talk about (technical) debt without recognising the commensurate leverage it provides to business. If a debt does not continue leveraging the business, then and only then, can it be considered carcinogenic. And the only reliable solution then is to simply switch the solution off or replace it.
It's easy to say this is a large company problem, but I've seen serious technical debt issues at companies that are less than a year old.
The easiest way to avoid these problems is to commit to clean code, and having the stones to stick to that plan. That means someone in power needs to agree with this way of doing things.
In my experience, the tone of a company is set from the top and works down. If the boss is non-technical and is not open to delegating, then there is nothing that can save that company other than new leadership.
You find another job ASAP. Organizational and culture problems will not be fixed by one new employee.
What happens is the competent people bail, leaving behind the less talented and/or less motivated which creates a positive feedback cycle.
> What happens is the competent people bail, leaving behind the less talented and/or less motivated which creates a positive feedback cycle.
Well described here:
http://brucefwebster.com/2008/04/11/the-wetware-crisis-the-d...
> I am new to the industry
That's painting with a broad brush, as software is used in a huge variety of businesses. But I'd plan on any organization without mega-income-$$$ (and even that is no guarantee) being pushed to their level of competence and beyond, having turnover to deal with, having features that marketing 'needs quickly to close a big deal', and maintaining zombie ware [it wont' die]. Welcome to my 15 years of experience...
> As with most things in software engineering the technical problems are symptoms of the economic causes.
FTFY. Seriously - #2 & #3, not enough money for a 'co pilot'. And for 2, at least there is some ostensible owner of the module, rather than a maintenance by committee which can be even shittier.
#1, maybe not cost effective to move, or would be opportunity cost. #4, I'm sympathetic to not shipping the prototype, but ever hear of first mover / time to market? obsolescence is one of the biggest risks in development[1]. Plan on shipping it and rewriting it in bite size chunks ... forever!
Anyway, it was a nice read. 'mortgage driven development' hah.
[1] In the Mythical Man Month, these are both mentioned, even though they contradict each other, and many more developers seem to only quote the 'don't ship the prototype'.
I nominate this author as the Fred Brooks of the 21st century. (Brooks wrote "Mythical Man Month"). These depictions of organizational sickness are spot on. I look forward to hearing more. The Carcinogenic Prototype is what Brooks might have called "the tar pit".