Most people think that making a website is very easy, just stick some bytes on a server and it renders some fancy buttons. But the software engineers for big companies are dealing with programming paradigms and design patterns that are so far removed from end user features that it takes a huge amount of work to develop the process and system that can be used to create a website like Twitter.
Physical Engineering concerns
Physical engineering such as making tables or train stations for example are typically a huge logistical design and planning problem. Compared with software where you just need a computer, you have concerns over sourcing materials and organising people in a physical space. These are obviously huge problems with many solutions that have been created. But these are largely limited to laws of physics which tightly constrain the problem. With the growing automation sector, the gap between physical and software engineering is becoming blurrier too.
Software Engineering, the philosophy trap
Press enter or click to view image in full size
When writing software, you typically need to setup your environment, what tools you want to use and how to deploy your software for others to use. This is pretty similar to physical engineering. But once you start writing code, you enter the philosophy trap, where you have to categorise objects based on what you as a developer think they are and how they should be organised.
Other fields of study that have this same problem for example are Linguistics and Law. Consider your definition of gender and how that has evolved, what a sandwich is, or at how politicians can accidentally create legislation with legal loopholes. Consider your left hand for a moment. Do you have one? How many fingers does it have? What classifies a finger? There are many different answers to this, but the general idea is “good enough” in physical engineering. In software, making a mistake becomes a bug, and this is why there are so many bugs and viruses in software. If you don’t cover all your cases, you can create something that can be exploitable or unusable.
The Design System problem
Note: This isn’t about feature design, but about tooling design.
The difference in law is that if something is exploitable, you just need to pass a new law that amends or overrides the previous and as long as that’s less exploitable, then you’ve got a net positive effect. In Software Engineering we build a design system on top of which we create features. This is a huge problem if someone new joins, as they need to learn the design system, which is usually pretty bespoke by project, compared to say some physical scaffolding which is pretty universal worldwide.
If you made an error in a fundamental assumption on how your design system works, or you need to do a migration, you need to make a huge change to your code. On a building site if you need to switch places with a different team, this is not that difficult, everyone packs up and shuffles over in a well coordinated way. This is less true in factories where shuffling the layout of a factory has the same issue. However again, a factory is constrained by physics, so solutions to physical movement follow similar patterns that are fairly universal.
What is an Object or a Module? What data model to use?
This is the fundamental problem that all software developers face. Following paradigms of Object Oriented Programming (OOP) which most people follow, to other paradigms like modular programming, there’s always a fundamental problem of classification. How should you group functions, what responsibilities should different parts have. And you need to make this decision for every part of your code.
Apps manage data, so there’s always a question of how to store data and what ways data relates to each other. Data can relate to each other in lots of different ways, but software sets out hard rules that you must use your data in a certain way. If you want to use it in a different way, such as join it with a different set of data, then you need to manually add in these connections at best, or at worst, you need to restructure your whole data model. For example, you can’t have multiple accounts say for your kids and you on an iOS / iPadOS device. It would be a huge undertaking to reorient Apple’s data model to support multiple users.
Unconstrained space
When you’re working in a relatively unconstrained space like Software Engineering, you have to reinvent the wheel a lot. You have to make key decisions on how the future of your codebase will look like without understanding the scaling requirements. This leads to a lot of developers complaining about “legacy code” and “tech debt”. In a physical space, rot and degradation force rebuilds and upgrades, but in tech, software keeps working, and this makes companies pour more and more incremental money into systems that aren’t designed for modern uses.
Don’t get me wrong, this is great if we can do this in some cases, but older systems can reduce support, such as for new devices like mobile phones or those with accessibility issues. And they require incredibly well skilled philosophers to decipher the meaning behind the code. Software raises the bar for how much research needs to be done for understanding the philosophical changes that people want to make in the future. This skills and cost gap is why developing nations usually grow through exporting physical goods rather than software.
Summary: Little capital, lots of knowledge
Writing software has the clear benefit that you don’t need to create construction sites and factories, but has the clear cost where you need to get a whole bunch of people to read through what can be millions of lines of what is effectively a research paper to understand the philosophy of the data, objects, functions and coupling to create new features, or more commonly, to alter the system to support more features.
This was a somewhat tongue and cheek post, obviously it can be incredibly easy to build extremely fast and powerful systems with modern software, but I wanted to discuss the problem for software developers when they start to work in bigger teams and companies that orchestrate large codebases.