A hierarchy of software engineering discourse
uvwx.github.io> I wonder if it would actually be better for software teams to build not the thing they want, but rather the thing that makes the thing they want.
Sounds so appealing to programmers, so much harder than it sounds. A dangerous fantasy many ships have broke upon.
The proposals of "yagni" and the "mvp" are in some sense a reaction to the danger.
On the other hand, when it works, it works. I'd say Rails is an example.
Rails is not an example. Rails was extracted from a single project codebase, Basecamp. Extracting a Solution Factory out of the solutions we make is a great idea, but starting with a factory-first mindset is what distracts and then breaks programmers.
Usually you have to first build the thing you want and then you can build the thing that will build what you want. You can’t go straight to the second. You need the experience of having built the thing.
Interesting, I keep building very similar patterns of automation in my systems management programs. Recently have started to think about what it would look like to make a generalized control software for the systems I interact with, to make the automation more uniform. I think this is probably what led to the teams that made ansible or saltstack.
The 'notation' level similarly seems to be promoting the use of DSLs, and everything you say here also applies there (maybe I'm jaded, but on more than one occasion, I have had to deal with the consequences of an incomplete DSL that had no real justification, and was clearly developed for egocentric reasons. I have also come across exactly one DSL (not mine) that was both justified and well-done.)
More generally, I agree with the author on the importance of seeing a hierarchy here.
We have a lot of progress to make here. Writing a language isn’t nearly as simple as writing a mere lexer, parser, type checker, and compiler/interpreter. You’ll spend just as much, or even more, time worrying about developer ergonomics (IDE integration, frameworks, toolchains, etc.). Projects like Language Server Protocol are great steps in the direction of making languages easier to implement, and LSP in particular is a great example of a “solution factory”.
That's intentionally making the problem three orders of magnitude more complicated than it should be, though.
In Lisp world, you start your DSL with a simple defmacro, and iterate from there. Lexer, parser, type checker and compiler is already handled for you, you only need to add some code where the rules differ. For a DSL you create within your codebase, you almost never need to do extra work here.
In ruby a DSL is just ruby as well, although it perhaps ends up having to _look_ more like ruby than a Lisp DSL would. (Not sure, only vaguely familiar with lisp). But at any rate it's the same thing as far as "create within your codebase", no extra work of a lexer, parser, etc. This is a thing people do in ruby and call it a "DSL", when they are using fancy techniques to make it especially concisely expressed and well-fit for the domain (but it's still really just ruby under the hood, no parsing step but parsing ruby code with your ordinary ruby interpreter).
Which has always made me think... so HOW is this different than an API anyway, why does it have a special name? Aren't all API's "domain-specific languages" of a sort? Certainly they can be good or bad APIs, in a variety of ways, including being a good or poor fit for the domain of work they are trying to enable. But if in the end the implementation is just (ruby) calling methods on instances, or (lisp) calling macros and lambdas (or however you'd describe it in lisp if i've gotten it wrong)... isn't that what every API is, what's the difference?
IMO, Calculator Construction Set is a better example (https://www.folklore.org/StoryView.py?story=Calculator_Const...)
Pinball Construction Set (https://en.wikipedia.org/wiki/Pinball_Construction_Set) takes that a step further, not only allowing you to change colors and patterns, but truly allowing you to change the pinball machine.
And of course, there’s tools such as dBase II, Excel, Visual Basic and HyperCard.
The thing that makes the thing I want IS the language simply put.
More completely it's the environment, architecture, language, requirements, customer, experience and team.
I've rarely found a need for a generator that wouldn't be more simply and universally solved via a configuration pattern.
Generating tests from a specification or operations from a schema don't seem to fall into this category.
Yeah it's the splendid old "hammer factories" rant from Joel on Software!
>A dangerous fantasy many ships have broke upon.
viz the yet-anothers of: make build system, programming language, configuration syntax, (mvc) framework, ORM, game engine, CMS...
Sourceforge and github are filled with monuments to unrealized dreams and wasted centuries.
> "1. Fixations... 2. Requirements"
Brilliant!
But I would have condensed the last 3 into one point: "Productivity."
Productivity - accomplishing more with less work - should be the #1 internal goal of every software team. Productivity means you have more hours of the day and more of your effort freed up. This means more time for testing; more time for enhancing your tooling; more time for spikes; more time for upgrades; more time to develop features. Even if no one does anything productive with the freed-up time, it still improves morale and emergency responsiveness.
You can of course go too far with it and ruin your fingerspitzengefühl by automating too much of your work.
Most teams have testers who are built-in advocates for requirements accuracy. But productivity has no built-in advocate. Developers themselves are the first to sabotage their own productivity in pursuit of - let's all just face it - fixations.
Seems like the spirit of this article is "don't debate test coverage, code design, etc. and just get the job done because meeting requirements is all that matters" and that we should focus on making code shorter. No evidence is offered that shorter code is somehow better than code that adheres to "best practices" or has higher test coverage. Has anyone here fit an entire enterprise software system into half a page, thereby obviating the need for unit tests and clean, understandable code?
This feels like someone that just wants to throw whatever code they want over the wall without being bothered to design it well, write tests for it, and adhere to a team's shared design guidelines. I can't stand working with people like this. Just my two cents.
EDIT: I see that this isn't the primary point of the article, and that maybe the author means we could rid ourselves of so much debate by creating more powerful tools and working at a higher level of abstraction. I think that some of these same things are being attempted, though, when someone suggests a design pattern or something within existing code that could raise the level of abstraction, and that it's worthwhile to have that debate rather than label code design best practices as trivialities.
I think it depends on the problem set and where one works, etc. There's a lot of use cases where getting it over the wall is perfectly acceptable and what the business needs. I can cite numerous incidents where a lot of code was design, written, and then out of circulation less than a couple years later. I also know of instances where some crappy code stuck and is in use 14 years later. The hard part is knowing which is what, and some of this can be alleviated by a decent modular design and taking on tech debt to refactoring/swapping out certain implementations for more robust implementations at a future date. It's hard to balance delivery time frames with robust/clean code, especially when so many opinions drive the later. Hopefully you get to work in a place that favors the slower, better planned approach approach.
> and some of this can be alleviated by a decent modular design and taking on tech debt to refactoring/swapping out certain implementations for more robust implementations at a future date
Absolutely, but even that idea of "modular design" would be something that needed to be discussed by the team - but this article would label this a "fixation". This distinction really isn't helpful. What IS helpful is knowing how much time to spend on such a debate, and when to set that stuff aside in service of moving on with the work.
> where some crappy code stuck and is in use 14 years later
I think this boils down to a roll of the dice. Its possible that some code is simple enough, or some engineer is smart enough, that tests are not really needed, but given the choice to not unit test would be made early, and its very hard to know the complexity of the solution at the beginning, you're really just rolling the dice with regards to whether it would work out. Sometimes you get a huge payout by delivering good code fast, but sometimes it can really blow up in your face with lots of prod errors.
Personally, I'd rather smooth out that large payout vs large explosion curve by just writing tests. If we could have pushed out the code faster, that is a loss, but if the code would have had many errors, thats a gain. So probability distribution with regards to payoffs is smoothed out.
"don't debate test coverage, code design, etc. and just get the job done because meeting requirements is all that matters"
I would say:
"Don't debate test coverage, code design, etc., unless they are relevant to meeting the customer's requirements."
Now, many times, these things are very relevant to meeting the requirements! Without tests, the odds of your code actually meeting the requirements is very low. Instead of finding and fixing problems with the code before it gets to customers, you get to fix it after you give it to customers and it fails, and you've probably just lost those customers.
To the "code design" point, I think that is covered in the "Models" section. Modeling the problem correctly leads very straightforwardly to getting the right design.
The problem is the only constant is change. Customer requirements are rarely static. So it's incredibly important to put thought into designing code that's also flexible and maintainable. Because what will happen is customer will inevitably thought of feature X or find bug Y and if they find out it takes a week to fix it, they won't be happy.
But I get what you say about doing those other things to make sure you don't run into the problem of changing requirements.
Part of the problem though is the varied skill level of developers and adequately timing the completion of these tasks. Most companies never budget appropriately for design and just expect banged out code. And there's not enough push back from developers on this, therefore it's become an expectation.
Maybe you didn’t scroll down? That’s only the second item in a hierarchy of 5.
Perhaps I shouldn't have it's the "spirit" of the article, since a larger goal for software engineering is defined with point 5 - regardless, the point is made that, even in the tools we work in today, these things are trivialities and are always a waste of time.
The idea of "notation" falling into this hierarchy is really fascinating to me. Typically, if starting a new project, I will select a language from those that I am already familiar with based on my comfort in the language and the performance characteristics of the problem.
The idea of the language you're using to solve the problem being one of the things you're developing is an exciting and scary thought. Exciting because you can immediately implement any ideas you have into the new language, scary because stack overflow won't save you if you paint yourself into a corner.
In modern times, I can think of two languages that seem to have been built to provide solutions in specific problem domains: Go and Jai. Go is a fantastic language in certain problem domains, and I wish I had more excuses to use it. Jai isn't available publicly yet, but the creator of the language builds games for a living, has built (and shipped) multiple successful games, and he's building the language so that games can unchain themselves from C++.
I believe the article of the linked article is correct that this should be part of more engineers' toolkits. It may feel like it's easier to just use keep using C++ (or whatever) but what feels easy now doesn't protect you from the inevitable problems of the future.
New DSLs/notations aren't conceptually different or more difficult than different libraries with differing APIs. In fact, each time you're coding to an API of a library, you are coding to a DSL - the API is the language. Writing real DSLs via code transformators/generators (e.g. macros in Lisp) just removes the noise caused by the mismatch between the API and the language syntax, making things potentially cleaner once you understand the API.
Documentation is still crucial, whether you're dealing with a library or a DSL.
For the last year or so, I've been fascinated by this concept, and I've been exploring language oriented programming with Racket. There was some discussion on this topic and the Beautiful Racket book here a while back [1]. That book is the best introduction I've found to the idea that you can build your notation to the problem you are solving.
Re: painting yourself into a corner, it is true that you can't retreat to Stack Overflow for help debugging your DSL, but the Racket documentation is some of the best I've seen, and common problems do pop up on SO.
You might be interested in our project (https://treenotation.org). Gone are all syntax characters like {}[](), which turn a 1D stream of tokens into an AST. Instead you simply write your source in 2-dimensional notation.
Makes it far easier to build new DSLs. (Evidence: see all the DSLs in GitHub.com/treenotation/jtree/langs)
FYI The only DSL with a readme is the poop one.
Whoops! Thanks for the reminder. We can get this fixed. https://github.com/treenotation/jtree/issues/63
> In modern times, I can think of two languages that seem to have been built to provide solutions in specific problem domains:
The obvious one to me is `R`!
Because the linked article inexplicably doesn't link to the Paul Graham article it references, here it is: http://www.paulgraham.com/disagree.html
I think that "code style, naming conventions, source code and repository layout" being commented on by younger devs makes sense since they likely don't have the social power or knowledge to make these other suggestions. Also those suggestions can be important in the early stage of a project to set precedence
I disagree with code style as a fixation. It's much better to have a uniform style if it improves the readability of the code.
I think the most important thing is to not make a fixation a 20 hour discussion.
I was thinking similarly, but rather that the process should be front-loaded.
If you are working on a team of engineers to produce a piece of software, and large amounts of time are being spent discussing a fixation such as style guidelines and linter exceptions then I would wager there was not enough (or perhaps even any) time spent on deciding and documenting those things at the start of the project.
If you find yourself having discussions on fixations repeatedly, it's almost certainly time to take a step back, as a team, and come up with the guidelines everyone can live with, and then, most importantly, document those guidelines.
This makes future discussions on fixations really quite simple: "It's in the documented guidelines for the project so either follow that, or propose a change to the guidelines as part of your pull request".
Yeah, I think there's a neglect to address that these fixations and best practices are meant to address collaboration and maintenance in a code base. Just cranking out a solution without regard to how others will have to maintain your code does not work so well on a team IMHO.
Are we focusing on making the next teammate's job easier and not focusing on delivering a solution quicker?
In an ideal world, I would prefer most every bit of code was written with your fifth replacement in mind.
This means that you've taken the time to document what you've done, test what you've done, and even included any relevant design and architecture concerns for why you have done it.
Without this information, your fifth replacement is likely to have many frustrating "why did they do that this way?" reactions which sometimes will lead to rediscovery of exactly why when they try to remove or change it and are confronted with the context.
It’s really bad to fixate on code style. That’s why you have a linter and autoformatter: so that it no longer takes up any cognitive space.
I'm not sure the "Environments" he cites are good examples for most types of software development.
Mathematica, MATLAB, Lisp, Smalltalk, are all development environments where there are a lot of questions around how to best deploy the software after you have written it. Maybe in some cases you have enough control over the customer environment where you can tell them "Just install MATLAB, then run this code we are giving you."
But what if the requirements are to deliver an iOS or Android app? What if you are building a web service? Or an embedded system? Or a game for a specific console?
Continuous integration and deployment are too very important problems that need to be carefully considered when deciding on or creating an Environment.
With Clojure, the answer is straightforward: here's a jar (or war), or some JS. It's no worse than mainstream languages like Java -- and much better, if you might also need to target JS.
Everyone likes to bash Lisp and Smalltalk for this, but what are you going to do with all your Python/Ruby/Java code if you need to deliver an iOS app? Or target an embedded system, or a game console? Every language has environments where it's difficult or (nearly) impossible to support. It's not unique to those funny languages that don't use curly braces.
Yes, and "environment" issues can certainly become fixations for some people. For example, VMs. It all comes down to the reason a person holds a certain view and how broadly that view is supported in their own specific programming sub-community.
I like the author's overall point but I think it breaks down in the very narrow way they define fixations.
If you are using Matlab you can embed the runtime and the program in an executable. If you target embedded you generate c-code. Both are $$$ addons and works quite well.
If you really want the customer to run some script "nativly" I would write it for Octave.
"What we want is a vehicle for exploring that space -- something like an IDE for your problem domain. It's been joked that any problem in computing can be solved with another level of indirection, but I wonder if it would actually be better for software teams to build not the thing they want, but rather the thing that makes the thing they want."
To this Lisp programmer, this sounds like plain old bottom-up design. I'm making primitives and syntax for The Thing I Want. My IDE essentially is an IDE for my problem domain.
It's a bit harder in other languages, but still more or less feasible. I think many people would be surprised by how much of my software is written in itself.
Most of these levels are actually concerns of different professions, which are today together comprise the art form called programming. For example coding will only become a profession when there will be very restrictive, agreed upon rules ranging from what kind of objects do we have (like service, dto, testcase, etc), how they look like, how they are named, where they are in the source code (with reference to the design of software), what is their maximum complexity, etc. Remember: a profession is a type of activity which have rules, which minimizes the possibility of mistakes. Which means there are rigid rules, killing creativity.
Software engineering is the blend of software, business and people. When the business is itself purely about software, and the sole team member is yourself, then I agree with the article.
The ultimate currency in software engineering is time. Time spent on ideation, time spent on requirement gathering, time spent on designing, time spent on communicating ideas, time spent on iterating features, time spent on maintaining existing codes.
If going through the hierarchy saves you time, by all means do it.
It's odd that he incidentally refers to "architecture" a couple of times but doesn't include that in the hierarchy. I'd put architecture above models because it defines not just how your models are partitioned and connected but how they communicate. Also, good architecture isn't an answer to here-and-now requirements but anticipates how an entire system might need to adapt to _future_ requirements.
If your structures are good, your architecture is provided out of the box.
Architecture is a word used by many people in software.
I ask the author of this article: have you looked at the definition of software architecture?
ISO, IEC and IEEE agree on what software architecture means.
Quality, as in code quality, is an architectural concern. Good practices are an architectural concern.
If you downplay the role of good software engineering practices as unimportant bikeshedding you cannot call yourself an architect.
"Requirements" is an often-misleading word. The danger is putting the customers on a pedestal while at the same time not really listening or understanding.
Supposed "requirements" are sometimes merely design preferences. Customers will often trade them for other things they want more (time to delivery, higher priority features, alternative implementations) if given a chance. Unless you have a good relationship with the customer so you can negotiate these tradeoffs, you may build something that meets requirements but isn't what they really want.
Very interesting.
I would have treated "requirements" differently - considering them as a mechanism to capture all levels in the hierarchy rather than a distinct level in their own right.
Disagree.
Need requirements to define what you want to achieve, before there is any point in discussing models, notations, etc.
How's that going for you ;)
Very well! Thanks for asking.
Where does "coming up with new manifestos" fit in? Because that's a demonstrably integral part of software engineering, too.