DSLs are a waste of time
leebriggs.co.ukWe stop calling the useful ones “DSLs” so this is a truism.
Is SQL a “waste of time?” Regular expressions, HTML, Makefiles, CSS (and CSS selectors aka jquery selectors)?
It’s the bad ones that are a waste of time. The ones still called “dsl” instead of just “language, “format,” or “syntax.”
> Is SQL a “waste of time?”
It certainly wastes a lot of my time because it is considered just a DSL and not a "real" programming language, and therefore doesn't get the same kind of attention towards making it better that "real" programming languages get. Maybe there is something to the original thesis.
Like, why can't I import shared query modules to compose into my queries? I can't imagine any other programming language without that feature. The execution engines support it in concept, allowing you to get there with some painful copy/pasting or using code generation (which seems to be how most people use SQL these days[1]). But because SQL is seen as just a DSL it doesn't get any attention towards making the language better.
[1] Which I'm not sure SQL is all that good at being the assembly language of databases either. It is strangely irregular.
I feel like the things you don't like about SQL are explicit design choices that make it well suited for many tasks, but perhaps not whatever you are trying to do.
It's like, "Geez, regular expressions sure do suck. They don't come with a package manager, concurrency, or a way to make system calls." I only feel comfortable embedding regular expressions or SQL in code because they are inherently restricted "DSLs."
If SQL was a general purpose programming language, then we wouldn't need SQL, we would just use existing general purpose programming languages.
Assuming you have some use case for query composition, why is the feature better handled within SQL as opposed to in the general purpose query language managing the DB queries and making the calls?
> why is the feature better handled within SQL as opposed to in the general purpose query language managing the DB queries and making the calls?
If the environment hosting and managing the queries (e.g. psql) is open source, maybe you can hack in support. Often that is not the case. Even when you are able, then your work becomes non-standard and thus not easily shared with other analysts, which kind of defeats the whole sharing of modules.
If you are writing a full-fledged application, using "real" programming languages, then a lot of people build their own query languages on top of SQL, treating SQL as just an "assembly language" that their query language compiles to, to add what SQL lacks. That's all well and good, but totally overboard when you just want to answer some questions about your data.
I do think you make a point that SQL tries to be too many things to too many people. No question, it is useful being able to embed SQL strings into other code without needing to worry about complex dependencies... Except you still have that worry as we already tried adding that composition through stored procedures, views, etc. which are dependent on a particular database state being true. It's there, just not very well thought out from a developer ergonomic perspective. Indeed, SQL is confused.
I think your examples are spurious and wrong, when compared with the specific kind of DSL the article talks about. SQL is a "Fourth Generation Language", much like Prolog, which is quite different to what is being discussed. HTML isn't a DSL; it's a type of SGML, a specific document markup language - you can't just shout "DSL" and squint a bunch and assume "All sort-of-languages are DSLs". Regular expressions, sure, but that's not relevant here because it's so universal - and mathematically backed (ever done a CS degree?) - that it's embedded inside every language now - i.e. this representation is a somewhat fundamental and mathematically general representation of interacting with strings - the DLSs in question here are, quite obviously, nothing of the sort.
The article is right. DSLs are, universally, awful, as they grow and grow until they encompass all features of a general programming language - but because of their evolution path the're awkward and twisted and lack basic features of a general programming language (and definitely lack all the tooling you get). Witness what happened with XML as a configuration language. What started off as another leaf node under SGML ended up - by way of the Java ecosystem - growing if statements and loops and all the rest, only in a horrendous, hard-to-read and harder-to-debug monstrous language.
So he's right. We should use real languages. And we should probably all just use Pulmi.
I think the problem here is more that people still need to work on their positions to figure out what they really mean, because while you have some points, regular expressions are the canonical example of a DSL. Their domain is strings, and they make certain things you want to do in them very easy to express.
The problem here is that while you can say “Terraform is more than a DSL” - if you ever back the layers or try have a conversation with someone who’s a true dyed in the wool Terraform zealot and start to try and understand where they’re coming from, you begin to realise they don’t love Terraform, they love this weird half language because they’re an expert in it, and they’ve built their entire career on being an expert on something you can only use for one specific use case.
That's from the article, and that really sounds a lot like talking to someone that knows and loved regular expressions (and I count myself as one of those people). The alternative, which is often done in lower level languages, is either some simple nested conditionals or a state machine, and plenty of people choose to just write state machines. I mean, that's all a regular expression is anyway, a shorthand expression for building state machines to process strings.
It should be no surprise that there are plenty of people that think regular expressions are wasteful and you should just write your own state machine (fewer than there were in the past, but they exist). Regular expressions are exactly like everything else that's being discusses in the article, but many people like them. Perhaps that's a point we should focus on to determine what makes a DSL successful and good compared to the alternative.
If you re-read my reply, you'll see that I said that regex is a DSL, but doesn't apply for different reasons.
I did read your reply, and I think your reasons for excluding it are insufficient. You can't just exclude any DSL that succeeds (has become "universal") and say "see? no DSLs are good!" The point of me noting how people resisted (and still resist) regular expression usage is to note that just like any other DSL there are those that dislike it and eschew its use and instead just write the code. It is exactly what the article is talking about, where it advocates writing in the base language and not using a DSL.
If you're going to make a serious argument that "DSLs are, universally, awful" then you're going to need to account for regular expressions a bit more carefully. That can be you admitting that you just dislike them and don't use them so consider them awful as well, but if you don't hate them then you may want to focus on the why, and it is likely a much more interesting conversation topic to pursue than "DSLs bad".
Regular expressions ... this representation is a somewhat fundamental and mathematically general representation of interacting with strings - the DLSs in question here are, quite obviously, nothing of the sort.
https://en.wikipedia.org/wiki/Regular_language
Regular expressions have a deep mathematical background. An ad-hoc DSL from a company that specifies infrustructure does not.
Regular expressions, as used in practice in programming are mathematically general for interacting with strings, but they are in no way fundamental. They are a DSL for generating NFAs and DFAs (depending on the underlying engine). There is nothing fundamental about them, they are just a shorthand (a DSL) for those abstractions. Indeed, the very article you posted notes that anything able to be implemented as a finite automaton is also a regular language due to Kleene's theorem.
> Regular expressions have a deep mathematical background. An ad-hoc DSL from a company that specifies infrustructure does not.
You very specifically went far beyond making any statement about ad-hoc DSLs from companies that make infrastructure. You said "DSLs are, universally, awful".
I'm also interested why you left out makefiles in your original rebuttal. I think makefiles are a good example a simple and useful DSL for accomplishing what it sets out to do. That's not to say it can't turn into a mess (what language, domain specific or not, is free from that concern?), not that it solves all needs to all people, but makefiles are simple, straightforward, solve a useful problem, are well known by many people (and easy to explain and teach), and I don't think most people would consider them a mathematical concept core to computer science. Are they awful? If so, why?
You're grasping at straws now. Just admit that regex is a much more fundamental technology than some ad-hoc, badly specified commercialware and we can be done. Go on, you can find it in yourself ;-D.
Sure, Makefiles are a DSL. Just a horrible one. They just make my point in a different way. Have you used a Makefile recently? They are AWFUL. Brittle, complex, half-bash, invisible-tab-prefixed, abominations, sent from another millenium. Let's take a modern codebase - say Python - you're much better to just use something like pyinvoke - handles all the CLI-level stuff, and you get the full force of the programming language you were writing in the first place.
You seem to have fundamentally misunderstood my purpose for engaging you. I tried repeatedly to move the conversation from a poorly supported absolutist statement from you to a productive discussion about what makes a DSL good or bad, but you're only interested in defending yourself and your over encompassing statement, not in actually having a discussion. Such as why does being fundamental in some way make it a good DSL? Are all fundamental concepts worthy of a DSL?
In any case it's clear you're uninterested in engaging on that and nor are you willing to consider your statement carefully and actually support it with additional examples (beyond some puzzling critiques of makefiles which make me think you don't understand the point of a DSL at all) or walk it back to a reasoned examination. So I'm not sure we'll get much out of continuing this.
I think you’re giving yourself quite a bit too much credit there. Most of this discussion you’ve been trying to argue about regex. We can drop it up to a higher level, sure, but you seem to only want to do that to avoid conceding in the specifics.
As I’ve quite clearly said, regex is a good DSL - so I obviously do think, and have argued above, that there is a split between “good” and “bad” DSLs, but examples of good ones are vanishingly scarce.
I think, given that it’s your position that DSLs are generally good; or at least that there are many examples that you find fine, then it’s upon you to provide examples. So far you named Makefiles, which are hilariously bad, but your response to my comments on it was basically some lazy ad-hominem, so I don’t hold out hope for any actual intelligent analysis.
But as I have outlined above there is a giant chasm of difference between mathematically-backed deep and broad DSLs, and… some ad-hoc commercialware that iterates towards being a full language but with bad, ad-hoc design and worse tooling.
The real issue here is a DSL without intention of enforcing limitations permanently.
A Regex, SQL, TLA, CSS and so on are all better DSLs than allowing a general purpose language because they are meant to restrict inputs of lower priority than other goals, where priorities are largely fixed.
A corporate DSL is meant to expand to whatever grandiose plans the company has for accepting impractical input from platinum customers.
Aren’t regular expressions the abstraction to state machines? They all get converted to a DFA or NFA, no?
Yes, and I said as much. All DSLs are abstractions to some sort of code that does some action. What I'm saying is that we have examples of bad DSLs, and good DSLs, and perhaps by focusing between them we can come up with some useful information about what distinguishes one from the other.
To me, that's a much more interesting (and useful!) discussion than just piling onto the DSLs are awful bandwagon, and I also think it's a flaw in any argument put forth in that argument that needs to be addresses before I can accept it.
Exactly. The languages in the article are nothing of the sort.
> DSLs are, universally, awful, as they grow and grow until they encompass all features of a general programming language
This is the Slippery Slope Fallacy writ plain.
A great many DSLs are stable over years or decades. By your reasoning, a stable DSL is bad because it's going to grow. Why is it going to grow? crickets
Peeling away the fallacy, we could perhaps agree that "the uncontrolled growth of a DSL is an evil best avoided." Or if I'm being less generous, we could look at the bottom of that slippery slope and conclude "general programming languages are bad," because that's what the author seems most incensed by.
General programming languages were meant from the start to be something like what they are today. They're intended to manage complexity, not pretend nobody will ever want to do complex things.
They are also not application specific. Language design is hard. DSLs are made up by people who's main project isn't the language.
Sometimes the motive for a DSL is something like "I don't like how this takes 5 lines of code in JS, if I made my own language it would only take 3"
The random "ever done a CS degree?" is weird, and the premise that regular expressions are fundamental and mathematically general and therefore not a DSL is flawed: ebnf is also like that, and even more general. It _can_ parse HTML. Doesn't stop it from being an arbitrary representation of abstract syntax patterns. It's a DSL through and though and so is Regex. Actually pretty hard to find that pure of a DSL, which such a defined domain, as many DSLs evolve to be more general language like over time as the article says.
Well, I remember them teaching that stuff, and regex is quite clearly a very different sort of DSL than the ones being discussed. That combined with the comment author's seeming lack of nuance made me wonder if they'd ever looked deeper than "all things without a for loop are a DSL". I think we are getting too far into the semantics here though, at risk of splitting hairs and geting lost in the details. The core point is that the article is talking about some quite specific types of DSLs, so you can't say respond with "No, DSLs are useful" and then list off a bunch of unrelated stuff as a rebuttal.
I was replying to a very unuanced article, with the headline “DSLs are a waste of time” and no further hedging on that headline in the text. So ya, I knocked down the argument on the argument’s own (very broad) terms.
No need for the personal swipe.
> The core point is that the article is talking about some quite specific types of DSLs
It used specific types of DSLs as evidence but I don’t see a narrowing of the claim itself? Where are you getting that?
> No need for the personal swipe
Fair enough
It used specific types of DSLs as evidence but I don’t see a narrowing of the claim itself? Where are you getting that?
What makes me sad, as I posted in a top-level comment, is that I read this exact same rationale 20 years go, by Steve Yegge, and yet we are still having the same stupid debate in programming about whether we should use DSLs and then the DSL grows and grows and, lo and behold, it's Turing-complete, but its ergonomics are dire and its tooling non-existant. And it's not like this is some niche part of the ecosystem - it's how most people deploy infrastructure.
What I despise is seeing the same problems come up again and again, and never seeing good solutions to them, jsut the same mistakes, repeated. Programming is, truly, terrible. Just a load of slaves building the Pyramids.
I don’t think it makes any sense to dismiss regex “because it’s so universal” — you’re disqualifying a DSL because it’s successful, then saying all DSLs are unsuccessful. It’s just a tautology at that point, same as the article.
(And SQL is not a dsl because it’s a … programming language? But also HTML is not a domain specific language precisely because it’s not a programming language? I don’t follow your logic at all. I don’t think either are programming languages, but both are languages in another sense of the word, as evidenced by their full names, and certainly dsls.)
I would be comfortable calling HTML and SQL both DSLs. Given the scope of their use.
Seems like the bad DSLs happen when one attempts to express a subset of general purpose imperative computation in an application specific way.
Untyped lambda calculus FTW!
It is. But nobody listens. Ever.
JavaScript was a cute little DSL once too!
I feel like the author missed a more obvious comparison between Puppetlang and HCL: vendor specific languages.
When you consider them from that perspective it’s clear that the decision is based more on which company you trust to serve your long term needs over any point in time implementation details. Is that company Puppet? HashiCorp? Pulumi? The implementation details obviously matter but if you’re investing 7+ years into an ecosystem like the author did, then there are a lot more factors than just syntax.
Yikes! JavaScript is probably not the best example of a “good” DSL. It’s a crap language that benefited largely from being the only game in town re browser access.
I mean how you define “good” is an endless discussion by itself. For the purposes of this discussion I think considering JavaScript a successful and useful DSL are sufficient. I share your opinion that it is by no means “good” by more mechanistic measures. :)
I consider JS a pretty great language these days, my only complaint is the lack of a batteries included standard library that does the stuff underscore et al do.
This is a good point but also a little circular since the good vendor specific languages tend to break free — like Netscape’s (and nominally Sun’s) JavaScript :-)
(and I think SQL started at IBM but I’m not sure if they tried to keep it proprietary or make it a standard)
Sometimes! Google still controls Go. Microsoft has retained control of dotnet. Java more successfully escaped Sun/Oracle, but that ecosystem is still governed by a very small number of mostly very large companies.
Regardless the author getting 7+ years of runway out of a specific technology is hardly a waste! My average technology switching time is probably closer to 5-6 years.
True! I think used Perl about that long, and probably Ruby after that. And those are (obviously) general purpose.
JavaScript wasn't really a DSL though was it? Not in the sense of this article. Brendan Eich was hired to embed Scheme into Netscape Navigator. That's not the same as the kind of crippled templating languages this article is discussing.
One is procedural (JavaScript), the other is declarative (HCL).
Both are (or at least were originally) “crippled” compared to “real” languages.
Both grew immensely each eventually escaping their single-vendor origins.
I don't think that this is a true comparison. JavaScript was limited mostly by its sandboxing, not what you could write in it. You can't really compare that to what HCL is and does.
It's like (speaking) languages really. The good ones are ones that have always been there and people have just sort of accepted into the mainstream (like English). Don't invent your own language. If you can do it in an existing language, don't invent your own.
SQL and CSS selectors fit the bill. Everyone has just kind of accepted them now. It's not an excuse to try creating new languages for every new thing, it's an uphill task.
If you can do it in an existing language, don't invent your own
then we should all be using c, lisp, smalltalk and perl (add your favourite old language here). because most younger languages don't bring anything new to the table.
The good ones are ones that have always been there
esperanto? not exactly popular, but it is definitely better than english and solves problems that english can't. (it's just that people don't realize the problem yet)
I guess I should qualify it by saying "Domain Specific Language" instead of "language". Newer programming languages are different because they have a huge body of extremely clever people working on it making sure it's up to standard. DSLs do not reach the same amount of rigor, they're just there. Some parsers loosely thrown together. Where's the documentation? The specification? The language servers, the editor support? None. Don't invent your own DSL (other than for yourself).
i am just picking your examples apart :-)
i think we both want the same outcome, that is well designed and useful languages, whether they are DSLs or not, and i agree that most DLSs probably aren't that. but not all general languages are getting that kind of rigor from the start either. some get it rather late after they became popular enough to have demand for fixing the problems that stem from their initial ad-hoc design. i am looking at javascript and php here in particular, but there are probably others.
DSLs, due to their nature are less likely to ever reach the level of popularity where its worth it to fix design problems. general languages on the other hand are more likely designed by people who care and without the pressure of a quick solution. (and its my understanding that javascript was a quick solution, which would kind of prove the point)
esperanto, btw, was designed with rigor.
My take is that all embedded DSLs are bad. By embedding your DSL in another language you might gain certain things for free, but you also sacrifice a lot of control and often extra boilerplate code is necessary.
The exception of course is Lisp where embedded languages can feel like standalone languages.
Stated in another way: standalone are sometimes bad, but embedded DSLs are almost always bad.
From what I’ve heard of and used myself LINQ works pretty well as an embedded DSL because it’s more like a language extension with some solid semantics rather than a classic embedded DSL that is not terribly far from transpilers
According to the article, the alternative to DSL is the programming language. So what the author understands as a DSL is some special-purpose language that is self-contained with its own run-time such that you have to string multiple of these to solve the task. Possibly, the author might have
The article doesn't discuss the possibility of building DSLs with the programming language and using them, all integrated together.
Yes, CSS is crap for instance. The web should have one language for everything in which styling is a DSL.
Regular expressions are better when they are in your language:
E.g. we can cleanly stick something into a regex's syntax tree without worrying about escaping:2> (regex-compile '(0+ (or "a" "b"))) #/(a|b)*/3> (regex-compile '(0+ (or "*" "b"))) #/(\*|b)*/ 4> (let ((x "[")) (regex-compile ^(0+ (or ,x "b")))) #/(\[|b)*/SQL would be my #1 example too.
It is probably the ultimate DSL if you are willing to get deviant with your tech stack.
it's kind of a catch-22, right? SQL is declarative. And while people seem to love declarative DSLs, they are generally really only awesome on the happy path. For SQL, sometimes, it can be incredibly obtuse when something goes horribly wrong (in terms of query speed) -- even with great tools like EXPLAIN. And a query that is fast on one SQL might not be so fast on another SQL.
For other declarative DSLs, I've definitely bashed my head against the wall and gotten a few gray hairs over "why the f* doesn't this goddamn thing that should work work". Sometimes it's because I'm not grokking how the platform works. Sometimes it's a bug, or unimplemented feature.
I suspect the reason why people don't absolutely abhor SQL is that the biggest players (MySQL, Postgres, Sqlite) have actually done a reasonably good job of making the 98% paths very good and 90% of the rest "good enough".
> the biggest players (MySQL, Postgres, Sqlite) have actually done a reasonably good job of making the 98% paths very good and 90% of the rest "good enough".
Exactly - Unless you are doing something really unusual (or wrong), the vendors of these engines have almost certainly encountered and optimized for some shape approximating your scenario.
If you want to get into some extreme ends of the practice, "ancient" engines like DB2 are some of the most capable. Many have been around longer than most developers today have been alive (myself included). That is a lot of optimization legacy to argue against. The halloween problem & iceberg meme comes quickly to mind. Why wouldn't you want to stick with something that has already dealt with all of that bullshit?
Eh, if you're at the point where you're doing stuff like "give me all records where a one-to-many has exactly two entries, and a different one to many has at least one record with property x" it gets VERY hard to do the correct performant SQL. I don't think that sort of query is too bizzare.
Sure. But you can create a declarative DSL without having to copy SQL.
> We stop calling the useful ones “DSLs” so this is a truism.
Bingo!
Lua was born as a DSL for configuration files. Today is way more than that.
JSON was born as a DSL for describing data, a less verbose alternative to XML. It is more than that.
JSON never got new features. We just made new things around it.
Which is why it's amazing for data interchange and obnoxious to directly interact with by hand.
It should be a rule that all mature configuration languages become turning complete
FWIW this wiki page I started may help frame discussions like this:
Survey of Config Languages - https://github.com/oilshell/oil/wiki/Survey-of-Config-Langua...
It puts a bunch of languages in 5 cateogires.
Terraform is at least not a YAML DSL, a very common pattern which I don't understand the appeal of. I guess the main appeal is that you can copy and paste, and pretend it's data in simple cases.
But even Terraform suffers from the problem of "external" DSLs -- you end up needing a general purpose language. And yes this pattern has repeated itself so many times.
Awk and Make both grew into general purpose languages, and in Make's case it's particularly bad.
Most SQL dialects have also grown a pretty bad stored procedure language.
So I do think making internal DSLs in general purpose languages is more promising. But they have to be designed for it. There have been many attempts to use Python as a DSL/config language, but it's missing a few things for that, and gets awkward pretty quickly.
Ruby does a bit better, but it's not perfect either. It's fairly big and heavy if you just want a config file.
>Terraform is at least not a YAML DSL, a very common pattern which I don't understand the appeal of. I guess the main appeal is that you can copy and paste, and pretend it's data in simple cases.
It's easy to write an interpreter for - everything maps to either a scalar, list or a hashmap in any language.
It also looks pretty familiar to most people.
It gets abused an awful lot but so does every language that gets popular.
I thing you're getting to the heart of the matter here.
I'm not sure if I'm being redundant, because I reloaded a few times and see that _many_ posts have been made in the short time since I started looking at this article and reading comments. But, the discussing seems to be bifurcating into different camps. Do you assume that the blog and discussion are narrowly scoped to sysop/devops problems, or a general diatribe on programming languages and product design?
For product design, a big part of the problem is choosing your audience. The blog author mentions being happy with a prior ops DSL because it let him think he's not being a "software developer". In my opinion, to target this type of user is to always be on the treadmill of toy DSLs. You have to over-simplify something to appeal to someone who wants to use it without deeper engagement in a programming/software development mindset. But then you have other users who hit the toy's limits and want more, and you either provide two entirely different products or start compromising the simplicity to add escape hatches. In the end, I think you'll always find that you need more of an API/framework in a proper programming language if you want to support all the use cases with one solution. Or you make a poor approximate of a programming language, quite likely repeating most mistakes in the ~50 year history of scripting languages.
There has been a lot of this churn in the sysop/devops space where system administrators are being given more complex and dynamic deployment problems. (Facing their devops mirror-world where programmers are being given more operations duties). This tension also exists in other domains, but I'd say the systems operations space has much more stark culture of "scripting" and "config files" as a distinct thing from "programming" and it shows in the tools and perspectives.
fully agree, especially with the second paragraph. even as a developer, i want a simple DSL that lets me declare things and not twist the data structures into code. and especially as a programmer, if code is needed for more complex structures, i want that to be done in a real programming language.
so i am favoring the two products approach, or maybe a language with a well designed syntax for data, where the relationship between the two is like that of JSON and javascript.
FWIW the JSON/JavaScript design is roughly where we're going with Oils
We have JSON support for literal data, pure data
But you can also interleave code and data in what turns out to be a very Lisp-y fashion, which is useful for configuration
https://www.oilshell.org/release/0.17.0/doc/hay.html
Slogan: We're adding the missing declarative part to shell, called Hay (Hay Ain't YAML)
That is, most YAML configs are just nested key-value pairs, where some values are shell snippets ... So I think it makes sense to add declarative data to shell instead!
It's easy to write an interpreter for
it's even easier to write an interpreter/parser for s-expressions.
i think the designers of YAML had the hope that it would not get abused as a DSL, which makes the abuse even worse.
and while i'd love to see an indent based syntax for data, YAML isn't it.
You can add other languages, such as Kotlin to that list. It is explicitly made to give you the ability to build a strongly typed DSL with proper scoping, references, validation, comes with all the benefits of a real language, including working language servers right out of the box that autocomplete everything properly. But that would require sysadmins to admit that editing all their config through nano with no syntax highlighting is an awful idea, no matter the amount of time they tell BUT WHAT IF I'M SSHING ON A SERVER IN ALASKA.
Despite all the complaints I have for Gradle (which, arguably, solves problems much more complex than "Ask AWS to burn through my cash"), its Kotlin DSL is impressively powerful and extensible.
ssh-ing somewhere strange on a small laggy connection is standard operating procedure. Not having internet is normal. Not being able to write gigabytes of crap into root's home is normal. Not to mention not installing some weird language environment like Kotlin just so the config is highlighted properly. Including all the bazillion libs i'll need. And all the editor configuration that i'll of course have to copy around everywhere. And then debug first, to get the language server working on old CrapHeadDistro 17.9 from back when Obama still ruled. Only to get chewed up in the next security audit for having installed too much useless old crap, because I need that to edit some config files...
You can pry vim from my cold dead hands. After pressing <Esc> thrice of course. You may even get me to install a syntax highlighting file somewhere and maybe use netrw if possible. But a random server you are adminning just isn't the same as your average developers single laptop.
>You can pry vim from my cold dead hands.
I won't need to, you're already giving yourself carpal tunnel with all the :wQ! INSERT-MODE :wmkGDonzn zz CWSFD, these hands are already marked for death.
actually the opposite is true. esc-meta-alt-ctrl-shift[0] is more likely to give you carpal tunnel, than vi where you switch modes and then type commands with single keys instead of twisted key combos. i actually get annoyed at how many ctrl-key combos vim has.
[0] i don't know if emacs is that bad, but that's a classic pun that just fit here.
I used vim for 10 years (not really an advanced user, but comfortable with it), and I just learned Emacs for the last 6 months...
I am now completely confused by the Emacs vs Vim religious war.
The two programs excel at completely different layers of productivity.
I can't think of any software that radically approaches improving your workflow like either vim or Emacs, and they are both wholly unique.
I also use evil mode in Emacs and while it feels like a great match, it has some impedance mismatch in some emacs packages.
It really makes me wonder if there is a deeper layer of integration that can be done here to really allow people to take the reigns off their interaction with computers.
I am now completely confused by the Emacs vs Vim religious war
i think at least half of those who keep up with the emacs vs vim war are doing it just to confuse newbies. the other half really just prefers the emacs way or the vi way.
either way though you are right. they are both very distinct, and it's not one being absolutely better than the other.
To be fair, I have never advocated for Emacs either. Nano for quick edits, proper IDEs for anything else.
every editing command in nano is either ctrl-key or alt-key. vi/vim is the only editor that i am aware of that doesn't make me stretch my fingers across the keyboard.
rclone allows you to mount remote fs via ssh, it has some quirks but overall I'm able to maintain my self host setup with this. also I'm sure there are plenty of similar plugins, maybe even officially vendored, for all the shiny IDEs
So, the author considers platform specification languages a waste of time, and has no idea what DSLs are (except that he knows about plartform specification languages)...
Yes, platform specification languages are a mess. Making them without the "language" part keeps all of the mess (the language is absolutely not the source of it). There are people trying to fix this, although I'm not sure if I'd place Pulumi on that group.
And DSLs are still a great software architecture technique with all kinds of applications.
DSLs are often the advanced step to a programmer guaranteeing his future employment:
1) don't document
2) obfuscate
3) pick obscure language
4) implement with your own language
I will grant that the most succinct representation of a problem is a DSL, but DSLs are inherent obfuscation to some degree.
Just because a perfect mathematical representation exists doesn't mean anyone besides the author will understand it inherently
This really rings true for me. I’ve used many DSLs over the last 20+ years. Build systems. UI definition files. Orchestration languages.
They always start simple and easy, but they always eventually grow so complex that using a real language would be preferable.
I’ve never seen a system remain at the same level of complexity (or much less shrink) over a multi year timeline.
This is simply because over times systems destructure.
Some developer eventually can't be arsed to stop and _think_ for half an hour about how to solve a problem, so proposes replacing the carefully thought out DSL with a Python script.
The typical programmer does have any respect for code they didn't write themselves or what it was trying to accomplish.
I don't think this is the root cause. Declarative languages really paper over details in many cases, and the mapping from the declaration to the imperative execution might not be obvious. When you're on the happy path, it's amazing (if it's a good DSL: because you can be confident you haven't forgotten details, that it's systematically correct, etc). When you encounter a regression, bug, or misunderstanding, in anger, it can be very very difficult to figure out what's going on.
Marcin here, one of OpenTF folks.
I think the redeeming factor of Terraform is not the language itself, though I really like the fact that it's not a programming language. But I think the redeeming factor is the tool that interprets this language, and the fact that this tool can be taught new tricks - one of which is the CDKTF (not a fan, I think we can do better than that).
Here's an analogy. Most folks I know are at least skeptical about Java. The same folks would then praise Kotlin, Scala or Clojure as "good" JVM languages. The relative success of these languages owes much to the Java ecosystem and the JVM interop.
That's the reason I find Terraform worth saving. It's the JVM and the Java ecosystem of our cloud world. We can still build great things on it, whether you like HCL or not.
This is a great takeaway from the article-- "programming languages are the better authoring model for cloud infrastructure". I was surprised by the simplicity of the "complex" terraform example the author offered, I have seen much thornier Terraform. The title, while appropriate for an article inspired by Terraform, I think misses the benefits DSLs have paricularly in LISP-derived ecosystems. A much better title for the article would definitely be "Terraform(ing) is a Waste of Time"
Well... it's part of the cycle. First you create a declarative language to describe WHAT you want the infrastructure planner to generate. Then when your customers discover its limitations, you create a pulumi style imperative "infrastructure builder" language (or library). Then when people complain they have to do everything by hand, you create a different declarative language using whatever transfer syntax is popular that month. Then you keep repeating the cycle to ensure you have something to talk about on your blog every funding cycle.
Though... yes... I've had decent luck with DSLs in Lisp-land. It's not a panacea, and there's a couple months of training when we pick up new engineers, but for our app, it seems to work out fine.
I like DSLs for many things but not for building out cloud infrastructure. It seems like the whole point of those things to build layer of complexity on top of layer of complexity. If any if those things meant you didn’t need to know all the thorny details of bash scripting, charsets, quoting and all, they might be a met positive but as they don’t you might as well write some bash scripts.
I think calling terraform a DSL in comparison to general purpose programming languages misses the point. If it used a declarative GPPL instead of HCL I wouldn't care, but imperative alternatives general purpose or not are 'a waste of time' & not fit for the domain, IMO.
CDK for Prolog would be quite the trip.
As it turns out, what matters is whether the model is declarative. As some uses of YAML go to show, declarative vs not is an orthogonal axis to how general purpose or otherwise a language is, and orthogonal again to whether it looks like a configuration file or code.
I hate yaml with a passion. It marginally better than xml for reading (wins huge on comment syntax) and worse for everything else. It makes zero sense we somehow ended up with it as standard configuration serialization format.
Note yaml is not a DSL. It's a tree serialization format! Everything interesting is happening after it is parsed. Extreme examples point to e.g. github actions conditions.
Anyway, back on topic - maybe not prolog for CDK, but still quite interesting: Dhall-kubernetes - https://github.com/dhall-lang/dhall-kubernetes
Better than making your own syntax for a declarative system, that now needs custom tools.
I'd prefer it to lisp, and to JSON if I have to hand read or write it.
What's the alternative, besides specifying your declarative stuff inside something like Python?
I agree. For some things yaml is good enough. XML, JSON and s-exprs are also good enough. Everything is good enough until it isn’t. At that point your configuration becomes code becomes configuration becomes code you know the drill.
My point is that the more complex use cases like GitHub actions conditions invent their own DSLs in yaml string values anyway because nobody wants to write in a half broken lisp but in yaml instead of s-exprs, so suddenly you’re writing yaml but with JavaScript. Might as well start out with JavaScript and not pretend it’s a simple declarative language when it clearly isn’t. CDK people clearly saw that, not sure if execution is optimal, but at least the general purpose tooling works (libraries, types, tests, IDEs, etc).
I'd rather we just stop trying to be unixy and support every workflow instead of just having a large but fixed set of things.
Like, GitHub actions could afford to crappy if you rarely had to deal with them. It would be better if you did in fact have JS...
But GH actions take multiple minutes to run, and a linter is near instant.
A less powerful solution would be saying "Here's some prefab instant actions that don't spin up a new container, just tell us what paths to include in for this linter and this formatter and click this box, and since we control it we can optimize it and fix the bugs and all that".
People invent DSLs instead of just exposing code often because they want it to be easy and expressive, but they make the easy things moderately hard, and the hard things near impossible.
But if the easy common things are just built in, you don't need any DSL at all, and you're not tempted to, because you just say "We cover common stuff already, if you want to do weird stuff we have Python for that".
But for some reason programmers would rather spend 10x the effort on a custom system rather than implement a specific common feature, because they want everything to be like a math equation, a description of something general.
Back in the early desktop era, adding stuff just because X fraction of users want it seemed to be all the rage, now people add things based on whether it fits with a vision or an idea.
CDK for Prolog sounds sick and disgusting. I LIKE IT!
I'm a little more familiar with DSLs w/ Lisp, but I built a simple make replacement with Prolog a few decades ago and it turned out to be useful. Replacing CloudFormation or TerraForm with a Prolog based DSL seems like it could be a win. I like Prolog much more than Pulimi in this role as you can succinctly and unambiguously declare dependency relations.
> I like Prolog much more than Pulimi in this role as you can succinctly and unambiguously declare dependency relations.
Prolog for Pulumi is an absolutely doable thing - and indeed sounds sufficiently diabolical that someone should do it ;-)
Or even Prolog w/ the AWS CDK. Hmm.. that might even be a business.
It's probably not a business... someone will just fork it ;-)
I could see a prolog based solution being quite powerful if used to its advantage.
Someone would say…why not hold your state in SQLite and make it queryable that way? And it would be fair. But nowhere near as exciting a problem.
Yes I absolutely wasn't saying declarative means not GP or vice versa, that was really my only point: I don't mind a terraform alternative that uses (or if terraform switched to) a general purpose language as opposed to 'DSL', if we can call the current incarnation that, as long as it (the GP language) is declarative.
Pulumi is an example that proves the exception: using a general purpose language, you create a declaration of desired state.
Think of it like if you used a GPPL to emit HCL. You get the power of that language: loops, function calls, unit tests, or even the ability to call external APIs. You're declaraing desired state and letting the framework sort it outb which is you say you aren't writing imperative code like:
if (!exists(dnszone)) create(dnszone);I didn't name it, but Pulumi's the main (multiple) GPPL alternative I was thinking of. I don't think emitting/compiling to something declarative makes it any better; maybe if you normalised that being the key checked-in thing, a bit like a lockfile, but which people really checked and cared if it changed, more so in a way than the source.
I want to write and review declarative, not compiles-to-declarative.
Hey, Pulumi engineer here.
The engine doesn't compile to a declarative template. It's a common misconception, however because the user's program runs concurrently with the Pulumi engine, fully dynamic resource graphs are possible, including those inexpressible using templates.
This is Pulumi pseudocode:
And in ~10 lines of code you have a script or webhook you can use in situations where SCIM would normally be required.const group = new ldap.DirectoryGroup("$saas-users); // or any other runtime query: for (const member of group.getMembers()) { new saas.User(member.id, { email: member.email, name: member.fullName, }); // could also do some other dynamic action, such as // calling a webhook to complete user setup(n.b.: this is a simplified, illustrative example that the program drives the graph using its own using loops or other control flow.)
Ok. I've never actually used Pulumi, I was responding to GP's description. Personally I want declarative for IaaC; some here are advocating for imperative generation of declarative IaaC (and claiming Pulumi as an example), which I think is barely an improvement (but nevertheless..) over imperative IaaC.
That’s the key. Declarative converging systems so much better than imperative and all the edge case handling necessary.
Sure. But some people don't want to do that.
Some people like to overcomplicate things but that's not a reason to care.
Sure. But some people don't want to do that.
> I was left with an expertise and knowledge of a language that was now, effectively useless.
VB did that to me; the lesson I learned is to avoid[0] languages with just a single implementation. As a consequence, I'm not very interested in languages like Ruby, Go and Rust. In the old days, every language had a train of compilers/interpreters following it. Pascal, COBOL, FORTRAN, even ALGOL. I believe that in those days, building a compiler for an existing language was a pretty standard hobby project. Everyone and his dog had built one, and manufacturers bundled their own ones with the hardware.
[0] Learning is hard; I jumped from VB to PHP.
Ruby has multiple implementations: https://github.com/planetruby/awesome-rubies
The main implementations are:
- MRI (aka CRuby, the reference implementation) - mruby (lightweight embedded engine) - JRuby (on top of the JVM) - IronRuby (on top of the .Net stack)not sure why you are making that distinction. sure, a few decades ago, a single implementation meant that a language was not widely used. but that is no longer true today. the biggest example for a widely used single implementation language is probably java.
and learning ruby, go or rust surely is not useless
The point, I think, is not how widely used it is, the point is that you become beholden to a single, often corporate vendor.
I'm not actually sure what made the transition from VB classic to the DotNET version so disruptive, and so damaging to VB in terms of popularity and mindshare ( possibly that C# was better while still sharing enough of the benefits? )
But from everything I saw it was. VB mindshare just fell off a cliff from everything I could see.
I mean, you could say similar things about the damage and disruption about perl->raku and python 2->3, which are both effectively single implementation ( ok, less true of python ), but the end result seems very different.
Perl5 is still nearly standard in linux/unix, especially anything not embedded or real-time, while raku is niche, although perl5 is more and more niche despite that, with many, many haters..
Python 3 is probably as popular or more so than 2 was
VB is an also ran in DotNET languages.
So, I dunno, I think there might be more to it that just "single implementation is risky". "Single corporate owner" might be a better one, idk?
ha yes, single corporate owner is certainly a different kind of issue. i agree with that.
The frontend ecosystem realized (at the time of React) that the (template) DSL isn't really the point. Its the declarative description of what the end result should look like (as well as applying the diff) which is the important bit, not how you generate that declarative description. The devops space is slowly moving in this direction too.
TypeScript is a great fit to model the unwieldy yaml / json schemas with precision, getting less in the way compared to most languages with sum types. Deno could be a great player in this space that avoids the complexity of npm, especially with its sandbox to restrict side-effects, networking and external commands.
CDKs come with their own multi-language compiler (jsii-rosetta) which results with a ~300MB npm install. Even when all you wanted was to use simple typescript functions to output some JSON and/or yaml.
This push and pull on "programming language vs DSL" will be with us forever. Some people love maven because it's a concise DSL, easy to understand if you understand maven. Some people like gradle because it's a DSL with a major escape hatch in that it's also just the groovy programming language with all that entails.
This dance has played out with the likes of the Javascript community. Just write a js script, no actually we need a pipeline so let's use grunt. Actually we hate grunt so let's use gulp, actually gulp is just wrong lets get declarative with webpack. Actually that's too inflexible and weedy let's just use a framework (like react-scripts) which just runs our code and gets out of the way.
After 10+ years of programming, it seems like we are on an endless loop of "let the framework do everything" to "let's enable maximum flexibility with a full programming language". (see ant in the java world for an even earlier version of this). It seems like the push and pull is this constant problem that "describing the problem is hard, and messing up that description makes everything more complex to maintain". The lack of flexibility feels nice because there's less to mess up, but then when something really complex arises you usually end up doing something 10x more nasty than you would with an ultimate flexibility framework. On the flip side, super flexible frameworks lend themselves to being completely unique butterflies that require a heavy up front research to understand (as, you don't know what that foo method is doing and why it does it). They can 1x the complex situation but it feels like the tendency is to want to overcomplicate simple problems by reinventing a framework for your company. (My company is dealing with this and jenkins pipelines)
> After 10+ years of programming
Ah - you're a young whippersnapper! I lived through those changes. I never understood why people thought it was a good idea to have a build-pipeline for an interpreted language like Javascript. But my colleagues did; grunt->gulp->laravel->nodejs->react, with a parallel succession of CSS compilers. That was in just the 5 years before I retired. It became a sort of Red Queen race; you have to run just to stand still.
I'm glad I'm out of it.
there are frameworks that can be used without a build-pipeline. but while i have been building websites that way for years, i only recently realized the significance of avoiding the build-pipeline. it's way more than just laziness
Ultimately, what is an API but a domain-specific language? This article could well have been written about that.
People often talk about Lisp as a language in which people specify DSLs, but that acronym was something I only encountered outside the Lisp community. The way it always felt to me was that I "simply" designed a set of datastructures and operations around the abstractions and metaphors of the problem domain and then manipulated them to solve my problem. Isn't that the essence of programming in any language?
It seems to be more of a spectrum in Lisps, spanning from syntax enhancements to full blown languages.
Yes DSLs have been a terrible failure in the infrastructure space. But that's just because DSLs are suited to a class of problem where the language of the domain is well-defined and changes infrequently. What we've seen with devops is an ecosystem that changes even more rapidly than frontend software development (in other words, a brakeneck pace). DSLs are great in any area where you want to allow less-technical or untechnical users generate some simple rules. So think finance teams using Microsoft Excel, or video game story writers using in-house DSLs to define game events.
What specific DSLs are you talking about being failures? Because this is absolutely ludicrous claim.
DSLs aren't a waste of time. But when you create a DSL, you limit the size of the community that will interact with it. So if you create a DSL, make sure that the benefit in expressiveness is worth the down-side of a smaller community.
The title is overly broad. They mean a very specific set of infrastructure automation related DSL
I tend to agree on that point. I think CDK and related approaches are much easier. I also find just writing SDK based scripts faster and more reliable. I find very little advantage in things like Terraform, despite my driving a few megacorps to adopt it. (Listen, in my defense, I couldn’t sell the idea of just scripting things, and generally they were repurposing admins who had no programming experience and had a mental block against learning a real language - but somehow a configuration file like language was ok)
However I’ve written a ton of DSLs in my life. They’re always useful, because I write them when I find it cumbersome to not. The broad statement about DSL is of course absurd. However at one point in my life I built a system that made it easy to write DSLs in a monorepo, and it was a huge mistake. Everyone wrote dsls everywhere and you constantly encountered code in some weird language that you had to reverse engineer. So, there’s a DSL entropy principle to be aware of - DSLs are useful if they are bounded. But like anything too much of a good thing ain’t
Hmmm. I just have to strongly disagree. I hit those hard spots in terraform, just like I did in puppet years ago. But I'd still prefer to use a DSL for configuration management (because that's what both are). Puppet actually managed to get to a very usable state by v5, if used with hiera, you could essentially define some types and fling data at them using 'create_resources', which is the equivalent of "for each" over a data set. If you stick to that pattern and make good use of hiera, your codebase will be compact and easy to reason about. The difference between your environments will be a few lines of yaml. In fact I use his with terraform, to that same end. I've looked at pulumi and I just don't like it. Terraform has it's quirks and limitations but I still prefer it.
Maybe DSLs are a waste of Lee's time because he knows they'll be "dead languages" when the next major evolution of systems takes place. But meh, I'll learn the next DSL, and maybe even hook hiera up to it too.
Steve Yegge made this same point ~20 years ago:
> The whole nasty "configuration" problem becomes incredibly more convenient in the Lisp world. No more stanza files, apache-config, .properties files, XML configuration files, Makefiles — all those lame, crappy, half-language creatures that you wish were executable, or at least loaded directly into your program without specialized processing. I know, I know — everyone raves about the power of separating your code and your data. That's because they're using languages that simply can't do a good job of representing data as code. But it's what you really want, or all the creepy half-languages wouldn't all evolve towards being Turing-complete, would they?
Via: https://sites.google.com/site/steveyegge2/the-emacs-problem
I'd rather they just fix whatever problem or missing feature caused me to want logic in the config file.
Ideally, why do I even need a nontrivial config file at all? I want more stuff that just works, and has standard interfaces to find and connect to all it's other pieces.
Even manually executing SQL to make a database is too much.
I don't want make files at all. They're only there because C/C++ doesn't handle it for you. Python doesn't need them. Node.js doesn't need them. They have build config stuff, but it's nowhere near as in depth as makefiles.
Apache is insane with all the different configurations, and how it's separate from PHP and MySQL and it all has to be configured to work together.
Lots of tools just work. They have few options, and all the important configuration options are set at runtime automatically based on input data, auto discovery, and things like that. Install and run. That's it.
If I need a full programming language to configure, then whatever I'm doing probably isn't configuration, it's more like plugin writing.
The point is that the config gets more and more complex, requiring more and more features, until you get a rather ugly turing-complete programming language. And the further point is that with Lisp, you don't need any of that. Code is data. Data is code.
But then you're moving towards less of an app with configuration, and more of a "Here's a new programming language and frameworks to write your own app!" Situation.
If possible I'd rather examine why people need so many config options, and design from the start so that things don't need to be manually configured as much.
If a turning complete config really is the way to go, I'd rather just use JS or Python, not something metaprogramming focused that just invites and almost requires building more layers on top to do real programming.
I think that you really have to use some Lisp to get the point here. The point is it's the best of both worlds, and the problem somewhat vanishes. But I don't know if I can convey that to you in words.
Over the years my opinion has grown in the same direction, but with a slightly different slant: domain-specific configuration syntaxes are a waste of time, and domain-specific processing instructions masquerading as a Turing-complete language are always inferior to a real programming language.
If you want to represent a data structure, PLEASE pretty please just use an existing data format (json, yaml, toml, xml). Those structures can be parsed and validated by generic tools, and you won't need to write your own domain-specific parsers and syntax highlighters to deal with a problem of your own making.
And if you want to allow for runtime evaluation of constructs, PLEASE pretty please just use a programming language that has friendly syntax (lua or python comes to mind). Don't get cute and try to invent your own language, just provide a good API on top.
Programming languages will always be better, more feature rich and more capable than any DSL. However, the whole reason things like terraform or puppet become popular is because the people who write the DSL don't consider themselves "programmers" and they don't want to be thought of as "programmers".
As somebody who identifies as a "programmer" and often has to manage infrastructure I would absolutely love for a serious IaS library to pop up in my favorite language. I would switch to it immediately. However unless we start hammering home that if you are regularly writing in these DSLs that you are actually a programmer and there's nothing wrong with that people will still flock to these tools.
Very much agree, I do wonder how can things like crontab be specified? The cron job syntax is its own fricking thing that I have to always look up. But then, I wonder, how else can it be specified? yaml file?
I've spent way too many hours this week looking for a better model for configurable timers than cron.
Cron is kind of awful, not just the syntax but the logic. You can't say "Every three days but not on Monday, if it would land on a Monday, defer that run till the next selected day".
There is also too many ways to say the same thing, so GUI editors are hard.
Yeah, basically it needs to be Turing complete but it falls short. Then why not just give us Lua or some real programming language. There is no real need for a cryptic single line syntax.
We have seen this trend in feature engineering. Some companies selling feature stores try to promote their DSL as the way to create features - instead of general purpose frameworks, such as Pandas, Spark, Flink, or event DBT/SQL.
In this case, DSLs may have a short-term wow factor, that you don't have to build an maintain feature pipelines, but you're always playing catchup when the latest feature engineering technique is not available in your DSL. And then, you have your developers careers to think of - do they want to put a DSL for feature engineering on their resume?
DSLs are more succinct. The are declarative. They are closer to English.
Perhaps if no relational database existed before today, the standard grammar would just be javascript - but I think not.
DSLs need not be declarative.
DSLs need not be succinct.
DSLs need not be close to English at all.
All of these are properties which may or may not apply to a _specific_ DSL, they are not defining features.
But they can be. They can have nice properties that general purpose languages cannot.
Why not?
Well the reasons for each point are a bit different so let's go through them one by one.
A declarative language cannot be general purpose by definition.
Succinctness. If you know exactly the domain you are talking about it means you can use many shortcuts and abbreviations that would be unclear otherwise. We all know this is true for natural languages, and it is also true for programming.
Close to English. In this case the reasons are more about aesthetics. English is not particularly good for general purpose computation as it struggles to express complex ideas with enough precision that it is crystal clear what is meant. A reader must apply some common sense to disambiguate. A general purpose language close to English would either be a Turing tarpit, or it would be a close-to-english language with some general purpose computation things bolted on.
> A declarative language cannot be general purpose by definition.
Prolog is both declarative and general purpose.
> Succinctness
life ← {⊃1 ⍵ ∨.∧ 3 4 = +/ +⌿ ¯1 0 1 ∘.⊖ ¯1 0 1 ⌽¨ ⊂⍵}
> Close to English.
We disagree that this is even desirable (and why English, rather than Hebrew or Dutch?) - but BASIC is a pretty good example of a general purpose language which is close to English.
Surely all programming is essentially the creation of a domain specific language, or at least a domain specific vocabulary that describes entities in the domain, the actions that they can perform, and the messages that they can send and receive. You can be fluent in Fortran yet be completely bewildered by a Fortran program because the vast bulk of the meaning is in a language that you have yet to learn.
As for being left with expertise in a language that no one uses any more, welcome to the club. My first language was Algol-60, the next was Leasco Basic, after that I wrote some Fortran IV, then whatever dialect of Dartmouth Basic was on the Exeter Uni time sharing system, then Z80 assembler, then 6502 assembler, followed by Turbo Pascal in its various versions (I skipped all the 'standard Pascals), more Fortran (but I forget which version) then VB4, 5, 6, then VB.Net then C#. SQL was involved for quite a lot of the time too. Then I retired and now I play with Nim.
Were all of those a waste of time too? I certainly have no use for the specifics of Fortran, Algol, Dartmouth Basic, VB.Net, any assembly language, or even C# now, but they were all useful at the time and I created things that were important and valuable for the companies in which I worked.
Oops, there goes the first 10 years of my career, where I used DSLs almost exclusively.
Ultimately learning a DSL is learning a syntax, a vocabulary, a model of execution, and the underlying domain.
The syntax shouldn’t take you terribly long unless it was badly made.
In my experience virtually any project whether you use a standalone DSL, an embedded DSL, framework, library, or writing something yourself, you will have to understand the underlying domain to be useful.
The model of execution can be simple or possibly tricky. There may be a small benefit to having an embedded DSL or framework over a standalone DSL as information learned could be transferable, but model of execution could still be more complicated and different than the underlying language in the other case. I’m thinking of things like Chef, Angular, and Parsec.
That leaves the vocabulary which you’ll either need to learn or develop.
You’ll take all of the domain knowledge, which as already stated, is the important bit. If you work in the same domain, even if it’s with a very different language, model, etc., you’ll have a great insight into what’s going on.
I feel like the opinions expressed in the article really just show a lack of experience of what happens after you’re done working with a piece of tech.
Nothing new or interesting in this article. I find the idea that a general purpose programming language is better for infra/config-as-code to be utterly laughable. No one is out there saying DSLs are “easy”. They are just far more constrained in a way that’s often useful.
His examples of confusing Terraform are not particularly interesting without some counterexample showing how his preferred CDK-style alternative is better (spoiler: it isn’t any less messy).
In fact, CDK/Pulumi-style programming is actually just an imperative-style wrapper that generates the underlying declarative IaC code. You might even call it a reverse DSL. But adding one more layer of abstraction away from what’s actually happening isn’t actually useful. It just adds more seams and more opportunities for leaky abstractions. Using a generic programming language to wrap your declarative infrastructure actually encourages more layers of abstraction. Not a good idea.
Could Terraform’s language be improved? Yes. But replacing it with JavaScript or Python is not the answer.
Buildah, specifically after struggling with containerfiles/dockerfiles nonsense, entrenched beliefs that agree with the author of the article. Dockerfiles are just an obnoxiously limited DSL that make all the worst assumptions (especially concerning layering), while providing nearly no flexibility. Buildah (when not using buildah bud) lets you write a container build script in the scripting language of your choice - most likely bash - and puts you in charge of layering.
I mean, sure, you don't get that automagic in-between later reuse - but I've rarely come across a project that does the magic incantations to get that to work anyway. Rather delegate the rarely changing stuff to a base image, probably updating in CI.
I agree. I wrote a long rant against DSLs here [0]. I won't repeat it here. This isn't an infrastructure DSL though, just a private DSL for our product.
Since then the situation hasn't gotten any better, but sort of stayed the same. Two people on our team have been solving bugs in this DSL for the last few months, rather minor ones. Tomorrow in fact I have a meeting with one of them about adding a sort of limited pass-by-reference semantics into the language (which currently doesn't allow pass-by-reference). I'm very torn about it, as I feel this is a waste of time, and we should ditch this DSL. Users complain rightly that it isn't a "real" programming language. Our "standard library" is very very lacking, for example there's no I/O.
We just added FFI 2 months ago. My team lead who was supposed to do this years ago didn't, and in fact effectively stopped developing it 2 years ago, feeling overwhelmed, and ended up leaving a couple of months ago.
What this language or rather ecosystem DOES do well is having an internal build system that is completely transparent to the users (unless there are bugs). Well, I designed this build system so I'm partial to it.
I had actually written the C# API which I wrote about at the end of [0], that was supposed to be a replacement of this DSL with a "real" language. However I got burned out a couple of months ago after my team lead left, and halted development. There was also a long series of heated discussions with some of my users, who wanted me to remove some of the (rather simple) guardrails I made, and give them the ability to build complex hierarchies above this API, an API which I designed for simplicity. I warned them those hierarchies would end up creating DLL hell and would make it necessary to build devops tooling and maybe even infrastructure as code [1], which IMO is silly for most of the internal projects, which contain at most 5-6 people. From 2 different organizational groups, but still. My users aren't developers per se, and have a more or less support role.
I'm still torn as to what the correct direction here. Long term our small team (only 2 full time developers and 3 part time) cannot support 2 half-assed non-standard DSLs, plus that C# API for our many internal users. Something gotta give.
[0] https://news.ycombinator.com/item?id=31205265
[1] There, I tied this response to the OP!
This is where Chef got it right over Puppet, but of which were written in Ruby. But Chef recipes are just Ruby files, and they make use of the Ruby language to do the same thing because it's perfectly capable of describing things that way.
This is mainly a complaint about over abstraction and intrinsic complexity.
Tools like terraform offer simple modules to make your devops life easy.
But simple looking modules abstract a complex thing by hiding details. Details which are critical once your infrastructure reaches a critical complexity threshold.
At this point the tool is a burden. It is actively getting in your way to understand and get at the the actual infrastructure you are maintaining.
This is why imo alternative like CDK got popular. It doesn’t try to over abstract for you. It just gives you an api you write custom code over. The complexity of your code goes up with the intrinsic complex of the infrastructure.
The AWS cdk is a pretty good example of how nice it is to have actual code doing the things. People know JS and know how to use it & it's imminently flexible.
By compare every dsl requires learning some new language. The languages seem to grow and grow. I know plenty of people with a little bit of Terraform experience, but almost no one with extensive knowledge. It's great that surfing along the top works as well as it does, and having such a confined language keeps a normativeness, but in general I so strongly feel it's just better to use available known well supported tools.
> Understanding how to handle complex data structures in say, Python is something which might help you fix bug’s in application code, you can’t throw a DSL at that.
True, but it's not an either/or situation. Maybe it's not optimal, but a lot of people do both. I consider myself a Python and Terraform expert and I'd say I reached that stage in Terraform in less than 10% of the time it took for Python.
I feel like I get the author's points pretty well. I had in fact chosen Terraform when it was newer as a greenfield project to deploy our infrastructure more quickly at another company. And I was happy to find out my current company had built all their infrastructure with Terraform, since by that point I knew it really well. But I still struggle with interacting with it, expanding it, and trying to get it to sing to me.
The example given is pretty good at demonstrating how Terraform itself has some unique problems that they themselves created: using counts for enabling/disabling resources, using locals as proxies for data comprehension, writing long-ass resource descriptions, often with built-in functions and ternaries to get it to do what you want in various scenarios. Trying to be DRY in Terraform can often be a chore. I think Terraform's MO is actually that some duplicate code isn't the worst thing, so you end up with piles of duplicate lines that are extremely difficult to read, and where writing a function would be super helpful. We had an engineer do a bunch of Terragrunt work for a project and I found that code nigh-incomprehensible.
So, I get it as well, I get where they're coming from.
I also think working with DSLs isn't a huge waste of time, since many of the things we work on are abstractions of abstractions. It seems like everything I'm doing today is probably going to go away in due time, but I know how to program, read docs, and pivot quickly. I do feel like as long as you're gaining an understanding of the underlying elements, your time spent in a DSL (assuming you're getting paid or you end up using it for several years), is still well spent.
They mention Puppet, and I remember getting a job offer once because I was able to read the Puppet documentation and write out a little script that could do the ask in Puppet, even though I had never used it in a professional capacity before. It was easy to pick up and I would've happily worked in it if I accepted the offer. The knowledge I gain around deploying to AWS or other cloud providers is often transferrable in that manner too.
The clouds themselves are a sort of DSL often, but I can't say that my time spent with Azure was a waste just because I'm working in AWS now. No, we delivered stuff on that platform, I learned its pros and cons, I gained some wider understanding of what cloud platforms could do, and I got paid along the way :)
Why don't we have a tool that lets us associate monetary budgets to concrete hardware specs, and then generates a cross-cloud (or cloud-agnostic) configuration?
budget: $23000
pricing: cheapest
cpu:
preferred: xeon
cores: 8
mem: 64gb
disk:
preferred: ssd
This is backwards. What you want is to specify resource limits and then have the maximum possible bill calculated for you.
If you want services to turn off if they exceed a threshold then you will need some sort of priority scheme.
A capped resource limit and/or budget does sound easier to manage. Here is your possible max; after that, shut stuff down in this order.
If it were any easier, it would likely be automated as a layer above Infra as Code (IaC). Someone may need to manually check on teams and ensure critical apps are not throttled before less critical ones.
Found a great option for you at $23000. Amazingly we managed to accommodate your budget. /s
It does sound foolish to share that directly to the cloud provider. They have an incentive to maximize spend--although they may also provide more than enough oomph with that much spend. It would lie on the payor to collect on all the available discounts (multi-year commitment, etc).
Assuming the tool is more or less one step removed from an elaborate spreadsheet and more of a crowdsourced, open optimizer, do you feel there would be any gains to generating multi-cloud setups, or just unnecessary complexity?
Is it possible to have a good-faith network of operators reporting monetary info along optimal routes?
Routes less like distances, and more like "given your compute and memory profile, a tier 2 storage bucket in AWS and 4gb VM in GCP would fit your budget."
This is a nice reminder of the importance of having your software be a "user-agent"!
Obviously you don't want to tell AWS what your budget is. You want your local tool that does work for your benefit to have a way to sanity check the configuration, and warn you if the only suitable VM at the provider turns out to cost a lot more than you expected.
Thank-you.
> local tool... to sanity check
Yes, exactly. Some open-source project as a layer between the cloud providers, and budget and other elements are just knobs to twiddle.
If this "open pricing" project could swarm various known constraints between money and virtual hardware, would it be a kind of constraint satisfaction problem?
Author is arguing that parsed DSLs are worse than embedded DSLs, and he's not wrong pe se - if you have a choice, you should choose parsod DSLs 99% of the time.
However, this is not at all an argument against DSLs. Using a "real" language to specify the cloud infrastructure, you'd still want your API to be declarative and ergonomic, i.e. you'd want a DSL.
Any api is kind of a dsl in the sense you need to figure out how to converse with/invoke it, but at least you can descend into the code to figure out what is happening.
A lot of DSLs are a wall to understanding what each "figure of speech" actually does. You need to navigate a parser before you get to the underlying execution of the language
Hyperbolic blog posts are so annoying.
Really, if OpenTF happens, you’ll be back to square one?
“ When the maintainers of OpenTF inevitably decide that they want to add enhancements that require language changes, you as the end user end up in a situation where you’re effectively back to square 1, learning (potentially) a new DSL.”
The post is from 2021
September 4th.
Oh no, I think I saw the copyright footer or something.
The alternative, using general programming languages, is a terrible idea. The last thing I want to do when dealing with a domain specific configuration is trying to figure the meaning out of hundreds of poorly written, badly abstracted, totally undocumented, lines of code.
Read the first couple of paragraphs, realized I have no idea what DSLs are or why I should care about them.
Define your abbreviations, people, especially when its not the first hit when googling it. It's not that hard and makes your writing much more accessible.
My favorite example of Terraform being a terrible DSL for infrastructure: code to find ipv6 addresses in a list of v4/v6 addresses
[for x in x.addresses: x if length(regexall(".*[:].*", x))>0]The problems with DSLs is usually poor ass documentation.
Throwing a full programming language in place of a DSL means now you deal with everyone and their mom inventing their own way of doing things.
People, just try pyinfra. Pure python for infrastructure as code. Ansible, terraform, salt, never more.
OC is arguing that Terraform is a suboptimal abstraction for infrastructure-as-code.
I have no dog in this fight. But I do have opinions about DSLs.
The declarative vs imperative, or some hybrid, debate is ongoing with build systems.
James Duncan Davidson, author of Ant build system for Java, famously wrote a post-mortem. Mostly wrt to choice of XML for Ant's syntax and semantics. TLDR: Wheels fall off a declarative model once imperative logic is added.
FWIW, Here's how I think about declarative DSLs. Such as a scenegraph.
You have an exquisite mental model for a scene. It's probably hierarchical.
You attempt to capture that mental model in code. The representation is probably a grove data structure. aka Directed Acyclic Graph where nodes can have key/value pairs for metadata.
You need to serialize that grove. So you pick a suitable syntax. Hopefully something that looks like VRML-97. Basically s-expressions (JSON without the syntactic vinegar), with some way to represent object prototyping (define/use), and don't forget identifiers ("id" field). Bonus points if you include path expressions in your syntax.
That (human readable) serialized grove is the declarative DSL.
In an future perfect world, your mental model, runtime representation, serialization format, AND the real world will be isomorphic.
Not isomorphic? Of course they're isomorphic. Why even mention it?
Funny you should ask.
In the Java world, dom4j for representing XML documents, and most of its progeny, is not isomorphic. Only JDOM2's class hierarchy design actually matches the serialization format. (As of the last time I checked, a few years ago.)
dom4j's mismatch, leaky abstraction, poor design, what have you, begat untold additional labor and heartache. Which is all completely mooted by JDOM2's correct design.
Ensuring correct isomorphism avoids errors in the same way that strong typing and garbage collection do.
To wrap this up...
In my experience, isomorphism is the exception.
I have no idea if Terraform is a good declarative DSL. But I do know that it should be isomorphic with the runtime representation AND the real world. And that should be settled before weaving in imperative functionality.