The Wetware Crisis: The Dead Sea Effect (2008)
brucefwebster.comIt's thinkpieces like this that make me believe that IT professionals specifically and white collar workers generally would benefit greatly from a mandatory 24 month stint on a construction crew. One of the most important lessons any job foreman can learn is that every crew needs at least one lame duck.
A single talented carpenter with an energetic but otherwise totally unskilled helper can accomplish more in a shift than two talented carpenters together. If you don't have someone to run to the truck for nails, hold shit, move ladders, etc. productivity suffers because your talent is wasting time on menial tasks.
Same thing applies to IT departments. The folks the author is busy sneering at are at least as valuable as the engineers they're actively trying to retain because (among other things) they soak menial tasks that would otherwise cut into their rockstar's productivity.
Hold on! The construction analogy doesn’t apply because in a lot of code bases, probably most there are very few menial tasks. Some code bases need IQ 120 + to do anything non trivial in and maybe even higher.
So there are menial tasks, but menial like “yeah just redraft this architectural drawing for me mate, to take account of the new 2021 fire regs, I got actual brainy stuff to get on with”.
Pairing senior and junior devs works. But not because the junior is an idiot, but because they have stuff to learn and raw talent and can figure stiff out. They probably already know how to code well from their passion or maybe bootcamps or interning. Not from university usually unless there is lot of coursework.
I am not saying construction workers are idiots. But I am saying the OP article is implying the worst employees are left by this dead sea effect. Are you saying you can absorb untalented people on work sites. In dev teams they slow things down it would be better to pay some of them garden leave.
I disagree with your point that this analogy does not apply.
You say there are some codebases that need over 9000 IQ to do anything. While that might be true, it is not the norm and a small fraction, so it’s not relevant to the broader argument.
You say it’s completely different in programming. Sure it doesn’t help if one of your workers is a complete idiot — but that was not the premise - it was unskilled but energetic.
And there are always menial tasks to do — you pointed out some example yourself.
So to sum up my points are: - most codebases do not require gigabrain for every single change
- there are always menial tasks to be done in a team environment
- “untalented” people don’t necessarily slow things down in development
- in fact talent is irrelevant when writing code - I would even argue that being talented is not always an advantage, in any field or profession - but that last point probably would need some more explanation.
100% of the most catastrophic IT disasters I've witnessed have been directly attributable to the most intelligent and skilled programmers I've worked with.
Example 1: I got hired to work in the dev department at a large regional printing company. One of the suites of software built in-house took large retail clients orders for printed store display material. From the client's initial order the software would automatically detail print batch sizes, assign these to the work queue, and would then go on to figure out how many of each type of display needed to be boxed and shipped to each of the client's retail locations and would create shipping labels, packing lists, etc. from this. At it's heart was a 10 page long SQL query that I couldn't make heads or tails of after two weeks spent studying it. I left the company certain of two things: 1. the dev that wrote that was one of the 3 smartest people I've ever met and 2. when he left the company they had literally zero chance of ever hiring someone to maintain that system.
Example 2: I got hired by the 2nd largest print media conglomerate in North America to help manage a fleet of 31 newspaper websites and a fleet of 125 niche verticals. This was at a time when the industry was just beginning to come to terms with their revenue models being gutted by craigslist and the shift to online marketing. To say the least money was tight. It was decided that the back-end systems that drove the integration between newsroom terminals, printroom layout qeues, and the newspapers' websites needed updating.
They hired #2 on my list of top 3 most intelligent people I've ever met to drive a complete overhaul of the system. It was decided a complete rewrite in a cutting-edge language was called for. The guy in charge didn't sleep for 3 days after which he'd provably onboarded and synthesized a complete understanding of the chosen language and it's ecosystem despite no prior experience, then went on to single-handedly rearchitect (feature complete) a system with roughly the same LoC as an operating system. In six weeks.
Over the course of the next 18 months the dev department responsible for implementing this masterwork (no sarcasm) floundered and the project was eventually scrapped having blown through the entirety of it's original 2M budget with literally nothing of use or note to show for it. Apparently nobody else on the team was capable of implementing and debugging the blizzard of microservices the new architecture called for.
My key takeaways: proposals to make codebases "interesting" should be met with deep skepticism if not outright hostility, and a roomfull of mid-level developers are significantly less likely to get an organization into deep trouble than a single rockstar with the bit between their teeth.
> Pairing senior and junior devs works. But not because the junior is an idiot, but because they have stuff to learn.
I agree so much. The "dead sea effect" or whatever is just a lack of knowledge transfer. This problem remains over time too, eg look at the use of Cobol in the banking sector. Newer talents dont tend to learn old stuff. So what else is left for management to do in the long term than document really good and train newbies. If you think "migration", consider the timing, when the lack of knowledge already becomes pressing.
To be specific on the garden leave: "[run to the truck] for some nails" is easy to formulate yet takes effort to do. The majority of the effort for most coding tasks is the formulation, and by the time you've explained to the lame duck how to do what needs to be done, and checked that they actually did it, it would've been easier to have done it yourself.
(were one to have suitable lame duck tasks, how would wetware lame ducks compete in an LLM world?)
This is where working at an actual construction site would be a good experience for you — or any developer really.
Run to the truck and get nails also requires explanation/experience to get right if you have never done it before — what nails?which truck?how many?etc.
It is completely the same thing. The first time around you might have to explain the menial tasks in detail - btw a good exercise anyway, since you need to be able to explain aka understand the task anyway, after all you need to review later. And the first time around there will be some mistakes or misunderstandings.
But it gets better over time. With everyone.
> Run to the truck and get nails also requires explanation/experience to get right if you have never done it before
After retiring from IT I volunteered at a raptor conservation centre. My previous experience counted for nothing and I was given numerous menial tasks: clean that water bowl, chop down that overhanging branch, sweep out that pond, re-attach that broken perch, put a trailer on the ATV and take this rubbish to the dump area, clean that aviary back wall, operate the music for the 11.30 display, etc.
After doing this for a while I had great experience of how the place actually ran, where things were kept, when it was safe to do things without interrupting flying, etc. I knew I was getting somewhere when my tasks started being looking after work-experience students, inducting new volunteers, helping in the displays, etc. It was actually quite humbling being a complete newbie again and having to go up new learning curves. I wish I had done it sooner in terms of resetting my objectives of what work actually involved.
You've just described a "junior"; I had mistakenly thought your "lame duck" was referring to the "salt" in TFA, who, on a useful timescale, do not get better.
Remember, for us the machine already plays the gofer role: handling menial tasks.
> Some code bases need IQ 120 + to do anything non trivial in and maybe even higher.
And in the majority of cases, I would say that this is "accidental complexity" - i.e. it can and should be otherwise, and calling out the unnecessary cognitive load, is a benefit.
> not because the junior is an idiot, but because they have stuff to learn
There's also value--sometimes hard to capture--in getting an external perspective on things. In a sense, they are a new user/customer for internal systems and processes, and can identify pain points that veterans have largely learned to ignore or bypass.
Couple of things.
1. The author said IT departments. This ostensibly includes everything from software development to systems admininstration, vendor management, and end-user support. Plenty of drudge work to be found there.
2. If your codebase requires someone to be on the edge of the bell curve of human intelligence to work on it is almost certain that mistakes have been made by very smart people.
3. The author is absolutely implying that the worst people are left, which is entirely my point. They have most likely never encountered the idea that managing folks to their strengths is more effective than trying to cram the room full of edge-of-the-bell-curve unicorns.
But you are mistaken -- each of us (the software engineers) already has numerous helpers who fit this description perfectly. The only difference is that we don't call them "apprentices" -- we call them compilers, interpreters, build tools and the like. Probably the only term that is commonly used on both types of helpers is "git".> A single talented carpenter with an energetic but otherwise totally unskilled > helper can accomplish more in a shift than two talented carpenters together. > If you don't have someone to run to the truck for nails, hold shit, move > ladders, etc. productivity suffers because your talent is wasting time on > menial tasks.Joking aside -- if you find yourself and your team spending too much time on menial tasks, you need to focus more on automating them. Of course, how much can be automated depends on how much hardware is involved in your work; as long as we're in software world, pretty much anything can be automated.
Maybe the "untalented" colleague is the solution to the marginal cost problem of automation in IT!
E.g. they won't do anything productive until CI and dev setup are "idiot-proof" - which is a feature! Noone wants to need arcane, private knowledge about how to fix that one problem with that one container in dev.
Alright, so how can I automate away daily standups, timecards, semi-regular random corporate/legal paperwork, and everything about dealing with devops team?
You can't -- but that's not menial work.
To me, it is, and/or it generates an infinite stream of one/off unique menial work.
This hasn't been my experience—the truly talented developers that I've worked with do more of the foundation tasks than the "salt" developers do. They're the ones who upgrade libraries when it's time, fix Jenkins when something goes wrong, and respond to a page and solve it, on top of the rest of their work. Meanwhile they're also automating the repetitive tasks in order to free up their own time and make it possible for them to move on to other things.
Automation is so easy in software development that your analogy to construction becomes pretty weak—we don't need someone moving around ladders because when we realize that's a need we just write a script for it.
I see y'all over there pretending lifers in your average IT department don't routinely make a career bunker out of legacy support among other odious pursuits despite the author explicitly pointing it out. In any case, if the department manager isn't tasking folks at their level of competence to extract maximum utility that's a failure on their part.
Yet I’ve worked in two types of places, the ones that had a lot of menial labourers and thus lots of shuffling of stairs and carrying of nail buckets, and in places that had very few to no menial labourers and just automated away the menial tasks or designed the systems to never need menial labour in the first place. If you want to keep the metaphor some plasterers use ladders, others use stilts. In any case plenty of ways to skin a cat and people adapt to the workforce available to them.
Brooks described the Surgical Team methodology, where one senior tackled the most complex tasks including design, the next senior people were helping with complex tasks and essentially "apprenticing" - with expectation that they will move on to become most senior person on a new team - plus a group of junior programmers took care of the simple "toil", while a group of supports (technical writers for documentation, tool makers, etc) together provided necessary cross-cutting skills.
This was modeled in the essay after "surgical team", with the most senior programmer being the "head surgeon" leading the operation, his direct assistants who could have major tasks delegated, and the juniors who could take on simple but still important tasks. (anaesthesiologist would probably be example of the cross-cutting support task)
> A single talented carpenter with an energetic but otherwise totally unskilled helper can accomplish more in a shift than two talented carpenters together.
I manage electricians, and this is highly dependent on the task. In most cases I’d rather send two JWs vs a JW and apprentice.
I’d send two JW to pipe a new feeder, but I’d send an apprentice and JW if they’re just pulling wire and the pipe is installed already. An apprentice can easily feed wire while the JW pulls, but a JW is going to be much better at piping than an apprentice.
I have spent all week exchanging emails with the "lame duck" in my ISP's accounts department. The task involved should be menial, just copy the bank account name from my email into a form, they are incapable of getting this right.
I grew up working construction on the side with my father, became an appprentice lineman, moved to aftermarket electrical installations in CNC machines, then ended up writing software products for over a decade. I've worked on a wide variety of software: Embedded projects requiring 2000 line assembly ISRs for legacy RS232 protocols, reverse engineering binary network protocols, implementing network server protocols from specifications, writing my own actor and dependency injection frameworks in managed languages, web application backend work, frontend work with JavaScript/TypeScript and frameworks, and so on. I also do a lot of product management, support, customer management, and internal management. Much of my customer support involves teaching external developers from large corporations how to use HTTP and SDKs.
I'm usually one of the smarter people in the room when doing software development and spend a lot of time helping others. In construction, I'm the enthusiastic idiot as the skillset is different. Most tech workers underestimate how high skilled tradework and many other positions outside their experience are, especially management work.
Having said all that, I feel uniquely qualified to offer a rebuttal. The article is describing a clear pattern that I see at large companies that are not focused on software development or lack solid engineering leadership. Ironically, the problem gets worse the more a company tries to outsource IT as they face all the same challenges with less control. It is an organizational health issue however, not some universal truth. If tech workers lack anything, it's experience with organizational growth, change, and group dynamics, not construction experience. Organizations are like people, they grow, change, figure out who they are, have identity crises and need different things in different phases. Ask the highest level/most competent manager you have access to for book recommendations if you're interested in this.
Anyway, back on topic, software "gruntwork" typically implies a department lacks the agency to automate away said gruntwork or lacks the skills to do so. As an example, I work with many organizations using the JVM, but none of them use Scala, Kotlin, Clojure, or any other "nice" JVM language. They use Java. In some cases Java 8. If you're writing Java, there is absolutely stupid gruntwork. I've written example applications for some of these organizations and I had to create code generation tools to stay sane in this environment. In software, you eliminate stupid gruntwork with tools, not people.
We do need average enthusiasts in software development, but it's not the same as construction. In construction the less talented person fetches things and does setup work. In software development, the less talented workers spend most of their time using libraries, plugins, frameworks, compilers, interpreters, databases, and languages while the more talented workers write them.
My experience isn't universal, and I'd be interested in hearing some dissenting opinions on this.
> Most tech workers underestimate how high skilled tradework and many other positions outside their experience are, especially management work.
Absolutely, (in particular) MEP (sheetmetal workers, pipefitters, electricians, plumbers) tradespeople and project managers are highly skilled/knowledgeable and the work requires a lot of coordination. You basically have to understand how an entire building/space is constructed from start to finish to properly plan and schedule a project.
For the record, plenty of other trades are highly skilled, but their trade/craft is more specialized/focused and doesn’t require labor from start to finish.
> as valuable as
I'd like to preemptively emphasize the distinction between "important to have" versus "marginal price to acquire", before folks start talking past one-another with different interpretations of "valuable."
Given that software has reuse and abstractions, we should all be embarrassed that there are any menial tasks at all.
A short stint on a Helpdesk will remove any illusion that menial tasks will ever be solved by machines.
The users are actually amazing and seeing what happens is awe inspiring - in a bad way.
I’m talking about in the technology function building systems.
This sounds intriguing, if it is true, there should be a name for this kind of observation / org law
Fuckin’ A man… fuckin’ A.
I actually have no idea what working construction is like, but I imagine it’s hell. I hear about jackhammering causing permanent damage to people’s nerves and all sorts of random accidents.
Having worked extensively in construction and IT (including a couple decades spent writing code) I have to say construction is a much more enjoyable and rewarding career. Pay and benefits are horrible though.
You get used to it much faster than you probably are imagining. I construction job on a decent project with a decent supe can result in a very satisfying workday.
After college, I lived with a friend who worked in the IT department at our college. He got called in on a Sunday once, and came back around lunchtime quite flustered.
"What's wrong?"
"Sam asked me what a traceroute was."
Sam was the longtime Novell admin. Yes this was a while ago...
"The worst part was, I patiently explained it to him, and he just asked me why that was useful?"
Just one example. There was more, of people of low competence who were very entrenched.
Novell primarily used IPX and gained support for TCP/IP very late, and then that still wasn't widely used even around 2003-5. Many networks used both: IPX for Novell, and TCP/IP for everything else.
Novell NetWare was most commonly used on a single, flat network with no routers or firewalls. The networks of the era were smaller, and the setup and routing of IPX is more automatic and simpler than IP. It didn't scale "up to the Internet", but it also needed less configuration and troubleshooting.
Notably, the "tracert" tool is specifically for IP, not IPX!
That's why Sam's Novell-expert friend hadn't used it and wasn't familiar with it.
It's like being surprised that someone that had used NoSQL for their entire career wasn't familiar with 3rd normal form and relational algebra.
One of the humorous tells of a novice is they haven't yet had to deal with all their tool knowledge being obsoleted and haven't had to understand that competent people might not know said novice's favoured tools.
The amount of random trivia in tech is a vast ocean. Nobody knows all of it. Specific cultures swim in specific waters, but out of habit more than anything else.
I knew this Sam well, as I also had worked in the IT department during my college days.
While you present a plausible sounding explanation for his lack of knowledge about TCP tooling, networking duties were under his purview as well. He spent a lot of time on the clock not hungrily learning technology, but sneaking off to smoke weed. I watched him once pack a bowl and smoke it while driving on the highway with his knees, coming back from a remote campus. He was very skilled at smoking weed.
Well, it was Novell. So not that entrenched after all.
It was longer ago than you might think. Novell was a big deal back in the 90's and early 2000's.
I'm a 'technician' in a company full of 'scientists', and I can say without question, the scientists need help, the technicians need knowledge, and neither of the two realms can really survive without the other.
Sometimes you need the guy who has burned his fingers repairing things, over and over, to review your design - he's going to tell you why you are going to burn your fingers. Other times, the guy doing things in a semi-glib fashion, needs to be pulled off the production line and enlightened as to why things have to be done a certain way.
To consider this relationship hierarchical is to weaponise the subject - instead, these are key relationships. A single great scientist with 2 or 3 good technicians can build amazing products.
3 scientists with no technician, rarely ever do .. but there is of course the odd exception where a technician, by making something really great, becomes a scientist, too - and scientists, without technicians and therefore having to get their fingers toasty, often make amazing technicians. (This is why the hierarchical subjugation of the subject must be resisted.)
Related:
The Wetware Crisis: the Dead Sea effect (2008) - https://news.ycombinator.com/item?id=23166786 - May 2020 (127 comments)
The Wetware Crisis: The Dead Sea Effect - https://news.ycombinator.com/item?id=22376468 - Feb 2020 (2 comments)
Internet Blogger A: "I'm super talented. Who are all these no-talent losers in every $BigCorp department I end up in? They take steps to entrench themselves and have no interest in innovating. The high talent people have more options and move on. Why can't I find a high-functioning place to work at?"
Internet Blogger B: "What is wrong with hiring process? We keep getting these clueless noobs showing up. They mess up every time we ask them to maintain prod. All they want to do is some random resume-driven development and then leave us with all the problems. Why can't we hire some real software engineers for once?"
You point a finger at someone, at least three are pointing back to you. And one to the ground, that's a different story so.
I've job hopped a lot over the years and now i make about 30x what i did at my first software job. Now I'm sticking around though despite some of the craziness that i'd prefer not to deal with mostly just to prove to myself i can stick around at a job for more than 2 years.
I’ve done much the same, but I’m curious if you have any other insights to share?
I’ve mostly settled into a place with a good pay rate-to-work expectations ratio, but I’ve also contemplated optimizing for pay alone.
i wouldnt say i have any particularly groundbreaking insights. Get things done, don't be afraid to learn new things as you need to. The best people i work with at any given time are the ones who are willing to try stuff and fail (but they probably wont because it seems like half of succeeding is just trying). So many devs out there get stuck because they are comfortable in their lane and are unwilling to get adventurous.
in terms of optimizing for pay vs lifestyle, i spent some years optimizing for lifestyle but im definitely in optimize for pay mode right now. Make a bit of a bag and get out and go back to optimizing for lifestyle. Things are better that way.
> The IT hiring process is broken. Amen. Not only is the IT hiring process broken in many organizations, the entire approach to IT is often broken. (...), Ruby Raley and I wrote an article for Cutter IT Journal that stated that an approach modeled after professional sport teams could well be far more effective, but no one has yet hired me to try it out.
Does this approach also apply to age? E.g., older programmers either take some higher-ranking position (management, product owner etc) or they get replaced?
[2008]
TL;DR: more talented people tend to move on to more lucrative gigs, leaving less talented people behind. The title draws an analogy to the Dead Sea, where water evaporates and leaves salt behind.
Really, that's all the article has to say. You're welcome.
I see what you did there... I guess one can write 7 seas of text that might one day be useful to a man looking to eat a fish.
Yeah that about covers it. I wonder if there's a term for metaphors that make something simple seem more complex.
I know it’s bad form to comment on articles without reading them, but my guess from the title was that this was allegorical to shepherds burning Dead Sea scrolls because they didn’t know what they were/how valuable they were to the historical research community.