No more DSLs: Implement and deploy a distributed system with a single program
catern.comIt’s a confusing article, especially given the emphasis on DSLs in the title, which I assume is a reference to k8s but I’ve never dived into it so ¯\_(ツ)_/¯
Building a distributed architecture is so much more than what’s presented here: error handling, consistency management, data plane, control plane, name services… this feels like it’s taking 2% of what Erlang offers out of the box and calling it done.
This article could use a disambiguation. Distributed systems are systems "whose components are located on different networked computers, which communicate and coordinate their actions by passing messages to one another"[0].
This article appears to be focused on running many components on a single computer under the same process tree. Perhaps a better title would be "No more DSLs: Implement and deploy microservices as a monolith".
No, this is distributed; see this line in the introduction:
>Functionality for distributed execution and monitoring is shared through libraries rather than by delegating to external orchestration systems, making a single-program system completely self-contained.
and this section in the examples: http://catern.com/caternetes.html#thread
The quote you pulled out mentions "distributed execution" being completely contained in a "single-program system" - isn't a single-program system, by definition, not a distributed system?
With the emphasis on distributed systems, I was waiting to see how the approach would help synchronization, replication, network dependencies, etc. But all the examples show calling other services as functions or spawning new processes. At the end orderd starts listening for requests, but I don't see any example in the article where the example program talks to another computer in the system. Perhaps I am missing something?
>The quote you pulled out mentions "distributed execution" being completely contained in a "single-program system" - isn't a single-program system, by definition, not a distributed system?
Nope. As the second paragraph talks about, there are other tools too (distributed languages) which also let you write a single program which is distributed. It's very common really - if you've ever written a shell script which performed some operations on another host with ssh, you've written a distributed program.
This is just another way to write a program which performs distributed operations, like using a distributed language or using ssh. (Well, rsyscall is the way to do that, this article is about an application of doing that, actually...)
That section is about threads. This is not a distributed system (running on different machines), it's a concurrent system (running in one machine.)
Anyway there is overlap between the two terms so strictly speaking the author is not wrong. But is anything with threads a distributed system? If so I built many of them, even with Python .
https://en.wikipedia.org/wiki/Concurrency_(computer_science)
They're distributed threads. As it says in http://catern.com/caternetes.html#thread:
>An rsyscall.Thread may operate on a local or remote host, or inside a container or VM, or on other kinds of nodes, depending on how the rsyscall.Thread was produced...
>All distributed operations are performed by method calls on rsyscall.Thread objects.
I don't understand which "DSLs" the author is arguing about.
Also, as other have said, the author's definition of "distributed system" (here, basically a "multithreaded" , concurrent system) seems to clash with the conventional definition (a system that has to involve multiple _computers_)
It is true that there is some overlap (having concurrency forces you to handle "some" of the problems with functional part of your program not being collocated on the same stack, you kinda have to introduce callbacks / promise / futures / actors / etc...)
But the moment you really have different programs o two machines (or , worse, two replicas of the same peogram on two machines), then I don't see how the single program would work / help.
Unless, of course, the "single program" is meant to be run as "one of the replica of the full system" ? Then I would love to hear the author's strategy for handling sharing info and workload between the replicas.
>Also, as other have said, the author's definition of "distributed system" (here, basically a "multithreaded" , concurrent system)
What gives you that impression? The article talks multiple times about using libraries to manage distributed execution, and shows those libraries in use.
Seems like a lot lower cognitive load and control. However, doesnt this effectively mean youre building lots of primitives from scratch?
Great for showing off how smart you are, not great for running real high load applications in prod with a team size over 1
Why do you think this? I can see a large team of competent Devs able to adopt this.
Competent in what? Writing bespoke systems management software? Who’s going to work on delivering value to the customer?
I am curious is there similarities to BOINC[1] ?
[1] boinc.berkeley.edu
No similarities. The focus is different.
- BOINC looks like an interface for end-users to volunteer compute resources for various (existing) distributed-computing projects.
- Post is about building your own distributed system starting from minimal-yet-effective principles.