What I mean when I say that machine learning in Elixir is production-ready
cigrainger.comI'm a big fan of Erlang, Elixir, OTP and the BEAM vm. However, I only have very lightweight server needs for my SaaS, so it just doesn't make sense to switch to it, especially as my code is mostly client-based/typescript.
If my core value prop would be around real-time messaging or streaming data, BEAM/OTP would be my first choice.
Slightly OT: I'm still in dubio about Cloudflare pages, but I'm sure that the platform for the backend reboot that we're starting will be in Typescript. There's so much advantage in having one language for everything.
I am really intrigued by the distributed systems capabilities that BEAM brings. I built a chat app using it and am also using Mnesia DB to store recent messages. However, my experience coding it has been terrible. I am using IntelliJ with an Elixir plugin and there is essence is no auto-complete. I used to write a ton of Ruby, but there RubyMine was really good at guessing what's available. Not so much here. On top of that more often than not the errors from Mnesia are just terrible. Most of the time I just get "TypeError" and a line number. Unclear what it expected to get and what it got passed instead. What IDE/editor have you found helpful? Are there any tricks to work around this? In Ruby I sometimes would use Pry Byebug to explore live what's actually available and iterate on my code in the REPL. Is there a similar approach you recommend here?
I've had success with the ElixirLS language server. It should work with most editors I assume, but I use it with Neovim. Autocomplete, diagnostics, goto definition work as expected.
I'm glad I'm not the only one who has had awful luck with that IntelliJ Elixir plug-in. Maybe I'm doing something wrong, but it just doesn't work for me.
I've been using emacs + elixir-mode + alchemist since Elixir came out and I have zero complaints, it all works beautifully complete with autocomplete, jump to definition and documentation popups. But obviously emacs isn't for everyone.
Vs Code with the default Elixir plugin is pretty solid. Not wonderful but you get all you need, including the debugger.
What do you use instead, out of curiosity?
Read dude
Have any of you used elixir-desktop [1]? It is a wxWidgets + LiveView bundle, pretty much like a Electron app.
In [2], Wojtek Mach explains how the team behind Elixir build Livebook Desktop. He explains how the project started, some subtle bugs found when building the app for MacOS, some limitations of wxWidgets in Windows, and many other implementation details.
It would be awesome if the Elixir team release something like elixir-desktop based on Livebook. That is, forking the Livebook repo and release an official template project for generating desktop applications based on LiveView.
Tangential to this, there's also https://native.live/.
Wanted to add to the usual "why is BEAM not more popular" conversation that there is also Gleam [1], which is an up and coming typed language on top of BEAM. It has the more common C-like syntax and a growing ecosystem of libraries, for example Lustre [2]. Unlike Elixir it has a much tighter syntax without all the macro magic. It also does not wrap around Erlang processes and you have to use those directly. To myself I see it as more of a Go for BEAM, although I am sure people would criticize me for making such generalization.
It sucks that syntax matters but it kind of does. I, for one, am somewhat put off by meta-programming and flexibility of Elixir. The idea of having to learn a syntax per-library I struggle to see the benefits of. I have never programmed Ruby or Lisp so perhaps I have not experienced the joy of what Elixir has to offer...
Gleam is a tooling development language, not a concurrency language. It runs away from Erlang's concurrency ecosystem. Also, its documentation, especially wrt concurrency, is sorely lacking. In its current state it would be a mistake it to use it for any distributed work, but perhaps this can change in time.
From what I understand, you get all of OTP as it is now – You only use the experimental Gleam packages if you want types. So you should be able to use them right out of the box, with the same level of guarantees (or lack of) that you get in Ex/Erlang. Still, I have not yet found a compelling reason to switch over – or at least try – Gleam.
Ah, I did not realize that. What is the point then? Why have a language on top of Erlang that "runs away from Erlang's concurrency ecosystem"?
Gleam has some third-party package(s) for concurrency, but I wouldn't consider them mature, and they wouldn't compare to anything like what Elixir has. It is not the primary or even the secondary focus of the Gleam language. I guess there is no point for me.
Quite frankly, I see a lot of text in this post and no numbers.
For something to be production-ready I'd expect you to at least cover major things like "latency to serve x in Elixir instead of lang y is k% better" or "EMFU we got when training x in Elixir was comparable to lang y".
These are two random metrics that are of course biased to my experience but the article just feels empty without numbers.
I see so much breathless adoration for Erlang and Elixir, how come it hasn't taken over the world in all this time?
Inertia: curly braces rule the world. Imperative/OO programming are ubiquitous, FP not so much.
Corporate sponsorship: even with curly braces and mutability, Go probably wouldn't have gained its mindshare without Google.
Performance: It's getting much better, but the BEAM was never designed for maximum performance. People don't like slow platforms, despite the other advantages.
Scale: much like one of the databases written in Erlang, Riak, you typically don't need a BEAM language until your solution is large enough that you've already written it in something else.
I love Erlang, but I'm a lost cause.
If curly braces rule the world explain Python? Some pretty well known apps are powered by Django.
Doing AI/ML without Python is doable but not quite as mainstream.
Sure, there are exceptions to every rule. Python is close enough to pseudocode that beginners can get familiar with the basic syntax easily.
I am not talking about beginner level software though. We are talking millions of users, very serious software.
> Performance: It's getting much better, but the BEAM was never designed for maximum performance. People don't like slow platforms, despite the other advantages.
> Scale: much like one of the databases written in Erlang, Riak, you typically don't need a BEAM language until your solution is large enough that you've already written it in something else.
These two seem to contradict one another. How can it not be fast but scale well (I'm thinking of a way as I type this but I'll let you answer)?
Latency vs throughput.
If you just want to do 1+1, 1 million times in a sequence, then erlang will likely be one of the last to complete the challenge.
If you want to do 1 million 1+1 in parallel, then erlang will get you there with minimal overhead, even if each number crunched will take longer then java, golang etc... the way it branches out is very efficient
When people talk about scaling they usually mean horizontally (more computers) rather than vertically (a more powerful computer).
Scaling like this tends to involve your system actually performing worse because, among other things, it has to go across the network.
You can do one thing fast, or many things slowly. For many common applications, doing many things slow is more important than off-the-starting-block performance
> How can it not be fast but scale well
soft realtime: Erlang gets slower with load, but keeps responding
fault tolerance: it's very hard to bring an Erlang application to a grinding halt
distribution: it's stupid easy to distribute an Erlang application and scale it horizontally
The reality is that it's not nearly as good as the loud proponents would have you believe. Performance isn't great unless you're comparing it to really naive applications written in extremely slow dynamically typed languages like Ruby or Python and OTP and real supervisor trees are not trivial to use correctly. Almost all of the Elixir systems I've seen have serious problems in their supervisor trees.
I've also repeatedly seen this idea from relative newbies that you can replace things like Redis with a simple Erlang key/value store, possibly using ETS, and the result is always much, much worse in terms of both performance and reliability. A lot of the older Erlang/Elixir proponents will tell you to just use Redis.
Most of the popular statically typed languages also have decent abstractions for concurrency and parallelism now while having far better runtimes, far better performance in almost all cases, far more libraries and much larger communities. Erlang/Elixir will never be more than a small niche.
This. I built my last company on elixir. Deployments of clustered BEAM in cloud VMs are far more complex than load-balancing across node or python servers, not to mention serverless. Any custom genservers we wrote always had stability issues. Type safety was a problem that caused crashes in production. Elixir is a really fun language and BEAM is powerful but it’s by no means a clear choice. My new company is on Next.js and it’s much easier to hire / develop / deploy.
It's funny how this works, I will never work with Next.js ever again (after doing it for years at multiple companies), I find it to be a total mess. Compared to how José Valim and Chris McCord (and others) lead Elixir and Phoenix and Vercel / Meta handles Next.js and React is night and day and I have close to zero trust in the latter. But then again, Next.js a tool to sell Vercel hosting and React has its own roadmap to cater to Meta, both of these are obvious perfectly valid and it's not a secret so people can make their own choices.
I don't know how many times I've done a minor update of Next just to have some undocumented internal change break everything, and reading their release logs are a joke, who thought it would be a good idea to just dump badly written git-commits and call it a day?
As an anecdotal warning, unless your team consists of TypeScript masters with extremely strict ethics you will end up with type safety issues too, they will just be harder to find (and if things go bad crash your runtime or client execution) :) Being easy to hire for has actually been a massive downside personally, it's very hard work finding people who actually know how to build a good React app, not to mention a Next.js ditto, but you'll get an near infinite number of applicants with skills "matching".
Jump to Elixir might be a bit too much. You could get strong type safety and scalability with ASP.NET Core without sacrificing productivity you are used to (it's also way faster than BEAM).
Here's one anecdote: I use Elixir professionally, for 8 years now. A few years ago I moved to the suburbs and met a neighbor two houses down who happens to run a software company based on Elixir with about 20 devs. Seems like I wouldn't be running into Elixir people in my daily, non-professional life unless there were a lot of them.
I have a theory about why there's more getting done with Elixir than you perceive. It's very productive, and you can get a lot done alone or with a very small team. You don't need as much devops. You maybe don't need as many frontend devs, because LiveView. And so on. Elixir teams don't need to hire as much, so when you look around, you don't see a ton of Elixir job postings. Fewer jobs discourages people from leaning into it as a career choice, which means fewer devs, strangling the overall growth of the community.
Also, at least in large tech companies, small productive teams are anathema to managers. Headcount is everything, and you have much more political power managing 10 Java devs than 3 Elixir devs. Never mind that the company is spending more on salaries, because that's the company's money, not yours. Anything you can do to have a bigger team is a win.
> how come it hasn’t taken over the world all this time?
I don’t know why it has not. However, it has such a strong set of advantages that people who know what it can do for them keep describing it as a “secret weapon”.
As others have mentioned, it will scale up on a single machine to make use of all the cores (unlike Nodejs, Python, or Ruby). It can already scale horizontally by clustering. Because of the way it is designed, I never have to define a Kubernetes liveliness probe for an Elixir app, whereas I have seen Dotnet and Nodejs apps freeze without crashing (cooperative async reactor can go into infinite loops; Nodejs is very bad about orphaning async execution, by design).
A lot of AI apps is going to involve connecting with unreliable third party services (for example, agents making calls to other api to get information or initiate actions), and may even be on hardware with unreliable network (IoT and edge) and this is where BEAM / OTP shines ahead of pretty much every other runtime and platform.
HN and elsewhere are riddled with Elixir developers extolling its competitive advantages for years … I have had very smart people argue to me why Typescript makes Nodejs so much better, but at this point, I have very little incentive to persuade them. Hence, “secret weapon”.
I take runtime pattern matching over TypeScript any day of the week, I work with data across boundaries, compile time checking means very little to me (and I'm sick and tired of the "await request.json() as SomeTypeThatMostDefinitelyIsNotTheRightOneButMakesForFancyAutoComplete;" that seems to riddle every TS code base). Sure, you can add runtime dependencies with the additional code bloat to have it runtime check types with TypeScript too, but you still don't get any fault tolerance, and you have to write explicit code to handle errors that according to the type shouldn't exist.
> it will scale up on a single machine to make use of all the cores (unlike Nodejs, Python, or Ruby)
Python definitely does "use all cores" on a machine with the multiprocessing package, not sure what you mean?
BEAM has a preemptive scheduler and being able to use all available core is a part of the standard runtime and language primitives, and does not require a separate library. The standard library, OTP, builds on top of those language primitives. The whole language and runtime is designed from its foundations to work with massive concurrency.
This is one of the things I think why Elixir isn't as popular: people think that Nodejs or Python can do with Elixir or Erlang do, but they don't.
I assume they mean Python doesn't do so by default (nor in a lightweight fashion). You can certainly use all cores with any programming runtime if you just run multiple OS processes. Indeed, that's how you implement multi-core on Ruby and Node as well. Although even then, the cores themselves aren't necessarily being fully utilized, even if you're ostensibly using all cores.
Because it is not by default, you don't really use all the cores by default either.
Python 3.13 beta1 can be compiled with no-GIL mode, which allows free-threading that could perhaps use all cores.
Let’s say we do that. How many threads can we create?
On BEAM, running 10,000 lightweight processes is normal. Phoenix is designed so that every incoming http request is its own lightweight process.
How does one manage that number of lightweight processes? The runtime’s scheduler keeps track of every single process so nothing is orphaned. It is preemptive, so no lightweight process can starve out another (though there are some exceptions).
They can also suspend cheaply, as such, works well with async io.
The closest thing to this are the new virtual thread feature of recent Java. I don’t think Java has the same kind of properties that will allow it to manage it as well as BEAM. There is a lot more to Elixir than being able to use all the cores.
Oh certainly; I never meant to think that Python competes with Elixir in this regard. It doesn't.
And if you have 4 cores, and 3 of them are blocked by IO, now you only have 1 core to answer requests. What happens if this core also stops in an IO? Your service stops.
That can NEVER happen in the Erlang Virtual Machine (the BEAM), because of the preemptive scheduler. This is only one of 100s of examples why the BEAM is the right choice for web systems where real-time can be soft real-time.
Nodejs will also move on if io is blocked.
However, since Nodejs queues bits of execution, rather than messages, errors can get lost. So then you have unhandled promise rejections… and then relying on a linter to look for those times you did not write something to catch the error. I was told this is a good feature of the linter.
Contrast that with the BEAM languages. An unhandled error crashes the lightweight process. Any linked process (such as a supervisor) can decide what to do about the crash (restart, or crash as well). We don’t even need liveliness probe (like in Kubernetes) because the supervisor is immediately informed (push, not pull).
You don’t need a linter to make sure it has a sensible default, because this is handled by design.
Nodejs, on the other hand, is error prone, _by design_, even though it does not block on IO either.
No amount of typespecing is going to fix that.
It's self-evidently false because you depend on multiple processes
I think we need to give it time. Python had a slow and steady growth from 1991 until today and it has eaten so much of the data analysis and ML world (backed by C++, C and more recently Rust).
But python doesn't vertically scale very well. A language like Elixir can grow to fit the size of the box it is on, making use of all the cores, and then without too much additional ceremony scale horizontally aswell with distributed elixir/erlang.
Elixir getting a good story around webdev (phoenix and liveview) and more recently the a good story around ML is going to increase it's adoption, but it's not going to happen overnight. But maybe tomorrows CTOs will take the leap.
> Python doesn't vertically scale very well
Strong disagree, this is a skill issue. I have written C++ modules for Python with pybind11 to speed up some code significantly but ended up reverting to pure Python once I learned how to move memory efficiently in Python. Numpy is very good at what it does and if you really have some custom code that needs to go faster you can run it externally through something like pybind11. It is a skill issue mostly. If you are writing ultra low latency code then you're right. You can make Python really fast if you are hyper aware of how memory is managed; I recommend tracemalloc. For instance, instead of pickling numpy arrays to send them to child processes, you can use shared memory and mutexes to define a common buffer which can be represented as a numpy array and shared between parent and child processes. Massive performance win right there, and most people simply would have never realized Python is capable of such things.
I said python doesn't scale well and you say "it does if you use an escape hatch to a faster language"
Sure. Writing C++ that utilises your resources effectively then writing bindings so you can use that in python is great. But with elixir, if I've got 8 cores and 8 processes, those 8 processes run in parallel.
If I want raw cpu speed I can write something in rust, c, cpp or zig and then call it still using elixir semantics.
Not to mention that with Nx you can write elixir code that compiles to run on the GPU. Without writing any bindings.
I went back and rewrote the systems in Python after learning more about how to manage memory more efficiently using common tools, methods, libraries
> Strong disagree, this is a skill issue.
I think the point with Elixir and Nx is that any difficulty in parallel performance is abstracted away
Just because it hasn't taken over the world it doesn't mean it's proponents adorations are baseless ?
I agree that the article falls flat on providing descriptive reasons how Elixir compliments machine learning. But that shouldn't be an argument against the product, but rather against it's cult following that does not provide sufficiently detailed and advanced examples and explanations. Something extremely common in tech journalism.
Good ideas and technologies do get underappreciated and wide spread adoption I would wager is not at all correlated to quality. Javascript, c++, java and others are good examples. Yes they have taken over the world at one point, but there are better designed languages out there. It just seems that people don't want to relearn a new paradigm and you can write software on anything.
> how Elixir compliments machine learning
one simple example: what Jose Valim calls Distributed²
You can have a livebook (similar to notebook, but distributed and live) and distribute your load not only on multiple GPUs on your machine, but on multiple GPUS on multiple machines
with basically zero effort
https://news.ycombinator.com/item?id=34546143
https://news.ycombinator.com/item?id=7277957
And also "taking over the world" is generally a larger task than people realize. C is still fairly strong, and at this point it is basically the baseline against we programmers issue our complaints, with only rare and very contrarian compliments given to it... yet there it is. Language turnover is really quite slow.
Elixir has a lot of promise on the surface but today I'm going to tell you why I am going to pivot my ten person company away from Elixir to a different language for two of our main customer facing services: typing and staffing. Elixir's typing situation is simply insufficient for complex real world use cases on teams with many developers and disparate services.
Elixir developers are also almost non-existent in the United States. It is far more productive and far less risky from a business perspective to hire someone that knows Python for the backend and is forced to work with mypy and pydanic using a major web framework.
The trope that functional programming leads to better code and is more maintainable is also a lie. Defect rate is just as high if not higher per module because of the typing problems.
What typing does Python have available?
Enforcing use of mypy is good enough for all real world use cases
As far as the lack of US Elixir developers, there is truth in that. It isn't that there are not Elixir developers, but that everyone who wants to work on Elixir is already working on Elixir.
Years ago, I knew this engineering manager whose strategy for recruiting Erlang developers wasn't to look for Erlang developers. He found generalists and polygots who would do well in any language, and were willing to learn Erlang, and recruited them, and then taught them Erlang. These are the kind of people who can take advantage of what Erlang/OTP (and Elixir) offers. They can do well with or without typing.
As for defects, the key isn't necessarily what's happening within a module, but how the various GenServer interact with each other. Having low defects within module boundaries are table stakes when working with Elixir. So if you're only measuring defects within module boundaries, you might miss the systemic problems related to interacting GenServers.
How is this different to typespecs and Dialyzer?
This is in the works right now: https://elixir-lang.org/blog/2023/09/20/strong-arrows-gradua...
See the first reply to the question linked below. I have nothing more to add.
https://elixirforum.com/t/how-to-make-dialyzer-more-strict/1...
OK, great, then I'll add that pattern matching on struct types is built-in and good enough for my purposes.
Haven't used the LSP-server in a while, but I think it runs the Dialyzer-checking by default too, so that layer of checking will likely be there without adding project dependencies.
"Best language wins" is not how programming language popularity at the top works. Java, C/C++, JavaScript all have/had various other reasons for their wide use. And of course there's huge inertia, even if a language has the requisite of other forces behind it for "taking over the world" it still takes decades.
But no need to put too much weight on this, lots of languages have more than enough users to be viable for a long time. The top stack overflow languages tend to have poor S/N in the user communities.
I think WhatsApp is still using Erlang and they are biggest messaging app in the world. Discord is using Elixir but it's more niche. I think if you look at use cases where they excel, you will find them there.
Discord uses Rust (and Elixir) https://discord.com/blog/how-discord-stores-trillions-of-mes...
They use both.
Same reason you don't see <Language people are enthusiastic about> taken over the world. The demands of the world are different to the demand of small passionate communities.
The demands of modern web platforms actually run really close to BEAM's strengths. It's still an interesting question but "enthusiasts are naive and wrong" isn't the answer.
I didn't say what you quoted
Because when you try to throw junior developers at it who had OO concepts in Python, Java, or C# drilled into them in college, switching to Elixir/Erlang feels like learning Martian. I had the fun experience of it being the second language I used in my career after PHP, and it definitely took strong mentorship and smart seniors who were really into Elixir for me to get up to speed.
At least when we used Elixir about eight years ago, we had to also learn Erlang as many libraries running on the BEAM VM that Elixir and Erlang share were only written in Erlang. So that made my brain melt further.
Once I got into the flow, I really liked it, and am glad I got the opportunity to learn it as a junior developer, but I've largely dropped it as there aren't many jobs in my area looking for Elixir.
TBH part of the BEAM runtime is actually pretty slow. is the problem.
Just because it is distributed cloud native doesn't mean for single requests or functions it is fast.* Want something flexible and easy and don't care about performance? Use Python * Care about performance? Use Rust, etc.Almost all SaaS applications are request based and do not need P2P or real time communication. BEAM is worse here. If a company does need that they can specialize that part of their application.
Out of the box, Phoenix applications respond to simple http requests in times measured in microseconds. What appreciable improvements from that do you get with Python? And considering how much of your total request time is not processing by the language (db calls, network latency, etc.), why would you decide on a language purely on the minor speed improvement of a small part of the overall picture? I’ll gladly trade what might amount to a few ms of request time for the concurrency model, scalability, and latency characteristics of Elixir and the BEAM.
To each their own, I suppose.
Yeah I don't get this argument either. It's like people are still stuck in micro-benchmark-land. Sub-1ms responses are the norm in Phoenix and I tend to never get anything above 100µs for LiveView messages, sure, they're noops, but calling it slow is .. well, strange.
Sure, you don't want to do number crunching in pure Elixir, but I'm always curious as to the actual needs people have, once being one of the "performance is everything" kind of developers myself.
For number-crunching there's Nx. As far as I can tell it's a solved problem, with easy handling of things like batching over clusters of GPU:s and things like that as a bonus.
As someone who was Elixir hire #1 at a company that eventually ended up with ~25 Elixir developers with a novel product that people actually used, I see people in this thread and elsewhere mentioning the unorthodox nature of the language or the supposedly poor performance, and I can tell you that none of that actually matters one way or another. The stuff that really hampers Elixir/Erlang adoption, in my own personal experience and from talking to friends who could have used it but went with other stuff:
1. From the outside, the benefits appear almost entirely theoretical, and in many cases, they are, and the issue with that is that in order to understand the real, non-theoretical benefits of having everything integrated into Elixir and OTP, you have to go all-in. Elixir has to own your app. Elixir is amazing, but when you start talking about its ability to do async IO, people say, "my language has a framework for async IO". When you start talking about its ability to automatically timeout and restart tasks and be fault tolerant, people say, "my language has a framework for network timeouts and I have systemd/k8s/whatever for application restarts". When you start talking about having ETS tables built in, people go, "I have Redis and it works amazingly well, why do I need it inside my app?" You can't adopt OTP piecemeal like you can with e.g., Tokio, or Redis, or Sidekiq, or whatever other thing. It doesn't really make sense to. Other languages might not be as powerful and not as well integrated, but through sheer force of effort, people make existing languages compose with general purpose tools and this is often "good enough". Of course you can write a single small service in Elixir, and this is fine and works, but this often runs into the issue of "now we have an Nth language beyond our normal languages, and we already have caching/monitoring/supervision figured out, so what is this actually adding?"
2. The library ecosystem is still very weak if you're used to e.g. Ruby, Python, Java, or Go (or even Rust, which already has way broader library coverage that is often significantly higher quality) and being able to find a mature driver/integration for a given API or native library. I don't know if this will ever catch up or reach a critical level where most people will find it acceptable. Stuff like the AWS library languished for, literally, years. Just critical stuff that's totally sorted in other ecosystems. This is purely anecdotal on my part, but it seems that way less people as a percentage of the whole community are working on libraries than applications. I have no idea why this is.
3. The performance - especially around latency and garbage collection - is great, but it's probably not "better enough" to actually matter in the domains where people tend to adopt Elixir. What I mean by this is, if you're already working on a project where the performance profile of Ruby or Python or Java is fine (web stuff, relatively simple network or CLI stuff), Elixir may be better in certain respects, but it's not so much better that it provides enough of a step change to make the other issues worth it. For our application, we had some very CPU intensive math stuff that dominated processing times, but we wrote that in Rust anyway, so the higher-level control layer could have been literally anything and it wouldn't have had any effect on performance at all. We picked Elixir because we liked it, but the product would not have failed for performance reasons if we had picked Ruby.
4. It is still, somehow, relatively difficult to find Elixir developers that understand OTP and the more operational concepts of the ecosystem (releases, deployments, observability etc.) at more than a beginner level. The community is still rather beginner oriented and there are a ton of folks who have done simple Phoenix apps and never gone further. I want to be clear that I mean this simply as an observation and not in a derogatory way at all. There is absolutely nothing wrong with having beginners in your community. Beginners are essential to ensure the community continues to grow and a language community without beginners is dying. But over time you have to build up a significant base of experts that can start and lead teams and companies independently, without having to rely on a class of consultants (of which the Elixir community seems dominated by - which is another conversation entirely that I will not get in to here). In Elixir, it seems like the proportion of experts has stayed small relative to the number of experts in e.g. Python, Ruby, Clojure, Go, or Rust. I can go out today and hire someone who understands the inner workings of Python as well as anyone working on the Django team. Anecdotally, as someone who has hired for Elixir roles, this seems difficult to do in Elixir, and I don't know why this is. I have been trying to understand why I have this perception and whether this perception is correct for literally years now, and I haven't been able to figure it out. (Strangely, Erlang on the other hand almost tilts too far in the other direction, with a ton of senior-level people and very few junior or mid-level.)
I could say more but it's probably better that I just leave it at that. I also want to say that I write all this as someone who really loves Elixir but wants to be honest about the issues that I have seen hamper adoption at the companies I've been at.
Great write-up! I've been using Elixir for a handful of years now, primarily working on solo/consulting projects and in micro teams. I'd like to think I have a pretty good grasp on the language and OTP fundamentals, but I'll admit that some the things you mentioned in your last paragraph about lacking knowledge of operational concepts resonated with me.
Over the years I've tried to learn more about releases, deployments, distributed applications, fault tolerance in environments with multiple nodes, and hot code upgrades; but none of the projects I've worked on have reached sufficient scale or complexity for those things to really matter. My impression is that it's difficult to learn about these topics without real hands-on experience (something that seems hard to replicate in small/solo projects).
Would you mind expanding on that last point? I'd be curious to learn more about these knowledge gaps that you observed, as well as any recommendations for "leveling up" past them, whether it be projects that one could build to learn more, books to read, etc.
> My impression is that it's difficult to learn about these topics without real hands-on experience (something that seems hard to replicate in small/solo projects).
Maybe. Unfortunately, I don't think there's a cheat code there. Often the single best thing you can do is to put yourself near people who know what you want to know.
> Would you mind expanding on that last point? I'd be curious to learn more about these knowledge gaps that you observed, as well as any recommendations for "leveling up" past them, whether it be projects that one could build to learn more, books to read, etc.
Related to my response above, as an example of something I observed, we were able to teach people who had little Elixir experience about simple GenServers pretty quickly. They learned the API fine, understood the client/server metaphor, started making contributions to the product/codebase, great, product managers are happy. The gaps became apparent when we'd hear stuff like "the thing I wrote is crashing, in the logs I'm seeing things like `exit: genserver timeout`. What does this mean? Where do I even start?"
The issue here isn't really "Elixir" per se. The issue is that Elixir was sold to them as this quick, pragmatic functional programming language (and it is), but what they got along with Elixir-the-language was a production OTP application that was actually a concurrent distributed system that they didn't realize was a concurrent distributed system and now they have to figure out how to debug that, which might have been more than they were bargaining for. Now OTP starts to dominate the conversation.
You say you have a pretty good grasp on OTP so this might not apply to you specifically, but if I could recommend a single thing to Elixir developers, it would without question be to read as much of the Erlang stdlib documentation and source code as you can. They're really, really good and highly readable. Don't be grossed out by Erlang. Read up on genserver, supervisor, sys, dbg, monitors, links. Understand what is actually happening when a genserver boots. Understand what actually happens when a genserver crashes, and how this appears to the supervisor. Understand when to use a supervisor vs. a dynamic supervisor. Understand when you might want to use the process Registry.
If I can recommend more than one thing, it would be to just build toys and provide your own hands-on experience. Elixir, being so dynamic, lends itself to discovery-through-play better than almost anything. Build a 3 node cluster on your laptop and play with rpc to distribute work. Build a little worker pool and randomly crash the tasks. Build something that dispatches work with Registry. Build a little network server that listens for new tcp connections. Kill random processes and see how things react. Use dbg or sys or observer or the other tracing stuff to see live state. Build a release and figure out how to remote shell into it. The only difference is that a real app would have a real domain, which you'd have to learn regardless. Anyway, I hope this is useful.
Thank you for your insights!
That was a super interesting write-up for someone who has a passing interest in Elixir but was curious about some of the issues you mentioned, thanks!
If you're feeling any less interested, I hope you reconsider and still give it a shot. There are issues, but there are issues with every technology. I promise you'll learn something.
It does look absolutely interesting but I feel like I'm already downloading too many books and trying to follow too many different technologies but I'll come around.
I feel like Elixir has a split personality: on the one hand you have libraries for anybody to use out-of-the-box with their specialty syntax and all (Phoenix, et. al.) but on the other hand Erlang/BEAM are heavy-weight backend development tools, things that I would imagine most people do not want to or would not get into. This came up on Elixir Reddit the other day, somebody was lamenting how their job uses Elixir, but a handful of libraries without getting into the OTP weeds, and the replies were: "why do you care if it works?". But I understand where the poster came from, I do not particularly care for Phoenix or Liveview, they are tangentially related to what I would like to do with the language, which is write robust backend services with minimal technical overhead. I am sure Phoenix is nice but and I hope to use it one day but it is not the selling point to me.
take a look at the Who's Hiring threads - despite the love the BEAM gets on this website, it seems very few folks are brave enough (or public enough) to hire Erlang/Elixir/Gleam/etc engineers. I've seen one or two Elixir jobs each posting and zero Erlang jobs in the last 6 months or so of Ctrl-F'ing the posts as I see them.
There's a pretty consistent stream of relatively senior postings on the elixir forums[1] as well as a number of successful large-scale elixir deploys (Discord, Heroku, etc) that probably do their own recruiting. I get the sense that YCombinator's technical mentorship focuses on language silos they have experience in, so this community is probably just a bit outside the "sweet spot" for BEAM hiring.
Hiring juniors (and getting work AS a junior I imagine) is a challenge. I don't think Elixir is that hard to learn (and has fewer foot-guns than many competitors), but it makes evaluation more difficult.
It sounds like Elixir might be a competitive advantage for attracting talent for the companies that do actually use it (much like python in the mid-2000s[0])
Elixir is a combinatorics mess (circa 2020: two different string types, three different types of exception, return AND throw versions for almost every func, slow compiler without much in the way of compile-time checking, constant breakages over trivial renamings "just because", having to burrow into erlang errs half the time since many elixir funcs are just wrappers). Half the libraries we used were abandoned. I take every recommendation with a grain of salt. I think people got invested in the hype, and are trying to keep the sinking ship afloat.
Erlang is a different story. If you're interested in the BEAM, it isn't terrible. It has a few of the same problems, but the combinatorics are reduced.
> Half the libraries we used were abandoned
People really need to unwarp their brains from how they judge libraries in Elixir compared to other ecosystems. Erlang is 30 years old. Elixir sits on top of that stability. Elixir will very likely never reach 2.0 because it doesn't need to. And if a 2.0 does come it will be simply to remove deprecated functionality.
Not having 12 major version releases per year means what you think are "abandoned" are actually stable and perfectly fine to use.
In Elixir, we don't really care about the last time a version was pushed. I regularly use and rely on libraries that haven't been touched in years because they don't need to be touched.
If the libraries worked, we wouldn't have had to play code archaeologist and find out that half our dependencies had been abandoned since 2017. Even something as simple as a JSON encoder--which is rock-solid in every other language I've used--had a number of bugs (this was four years ago, so memory is hazy, but it had something to do with ambiguity between arrays, lists, or tuples) Erlang is stable, but Elixir sure as hell isn't. Back in 2019, it seemed as though every point release brought a slew of breakages--and usually over the most trivial and pointless things, like adding an underscore to a built-in for "consistency". I've been developing for over 20 years, have used all of the mainstream languages, and Elixir was the absolute worst.
> Even something as simple as a JSON encoder--which is rock-solid in every other language I've used
JSON will be built into the next release of the BEAM: https://erlangforums.com/t/erlang-otp-27-0-rc3-released/3506...
How are there three different types of exception?
As for "two string types" maybe I'm not working on hard enough problems, but in ~4 years of Elixir I've never once needed to use a charlist. My understanding is that it's a backwards-compatibility thing from Erlang and I'm not even sure when I'd ever need to use it over a string.
raises, throws, & exits. rescue vs catch. as for binary vs charlist, some dependencies wanted binary, others wanted charlist. faced with the choice of re-write the dependency to use binaries, or wrap it with pre/post converters, we chose the latter. after the third or fourth global search-and-replace thanks to elixir's renaming of built-ins just for the hell of it, we re-wrote it in go, and never looked back.
We’ve had Elixir in production since 2017 and have not found any of the items you have mentioned to be issues.
- two different string types: You have undercounted (three types in Erlang, and Elixir adds a fourth), and suggested that something which is a non-issue for the vast majority of Elixir code. Most Elixir code deals with `String.t()` (`"string"`), which is an Erlang `binary()` type (`"string"` in Elixir, `<<"string">>` in Erlang) with a UTF-8 guarantee on top. A variant of the Erlang `binary()` type is `bitstring()` which uses `binary()` to efficiently store bits in interoperable ways. Code interacting directly with Erlang often needs to use `charlist()`, which is literally a list of integer values mapping to byte values (specified as `'charlist'` in Elixir and `"charlist"` in Erlang; most Elixir code would use `String.to_charlist(stringvar)` in the cases where required.
Compare and contrast this with the py2 to py3 string changes and the proliferation of string prefix types in py3 (https://docs.python.org/3/reference/lexical_analysis.html#li...).
- three different types of exception: true, but inherited from Erlang and the difference is mostly irrelevant. The three types are exceptions (these work pretty much as people expect), throws (non-local returns, see throw/catch in Ruby; these are more structured than `goto LABEL` or `break LABEL` for the most part), and process exits. In general, you only need to worry about exceptions in most code, and process exits only if you are writing something outside of your typical genserver.
- return AND throw versions for almost every func: trivially untrue, but also irrelevant. Elixir is more sparse than Ruby, but still comes more from the TIMTOWTDI approach so most libraries that offer bang versions usually do so in terms of one as the default version. That is, `Keyword.fetch!` effectively calls `Keyword.fetch` and throws an exception if the result is `:error`. It also doesn't affect you if you don’t use it. (Compare the fact that anyone who programs C++ is choosing a 30–40% subset of C++ because the language is too big and strange.)
- slow compiler without much in the way of compile-time checking: I disagree with "slow", even from a 2020 perspective, and "compile-time checking" is something that has only improved since you decided that Elixir wasn't for you. Even there, though, different people expect different things from compilers, and not every compiler is going to be the Elm compiler where you can more or less say that if it compiles it will run as intended. (I mean, the C++ compiler is both slow and provides compile time checks that don't improve the safety of your code.)
- constant breakages over trivial renamings "just because": false‡. Elixir 1.0 code will still compile (it may have deprecation warnings). To the best of my knowledge, nothing from 1.0 has been hard deprecated. ‡If you always compile `--warnings-as-errors`, then yes you will have to deal with the renaming. But that is a choice to turn that on, even though it is good practice.
- having to burrow into erlang errs half the time since many elixir funcs are just wrappers: not an issue in my experience, and I can only think of a handful of times where I have had the Erlang functions leak out in a way where I needed to look at Erlang error messages.
Elixir isn't suitable for everything, but frankly your list of so-called shortcomings is pure sour grapes.
I guess I will just have to stay sour with my sub-second compile times, actual compile-time type checks, stable naming of built-ins, single-binary deployments, and uniform error handling.
Better to say that whatever your resulting platform is (probably Go based on what you’ve said), it suits your constraints.
We've found that our constraints are better met for our most important project by Elixir. We are building something else in Go (mostly because it will be easier to find cheaper contractors to build these boring things once we have the first version done) and we still deliver a number of things in Ruby, and there's Typescript for some things. Use what works at the time you need it.
I personally find Go to be a tedious language and think it could use a fair number of DX improvements, ranging from adopting some form of Rust's terminal `?` sugar to just better recommendations on project structure and layout. It took far too long for Go to get generics, and the platform-specific build files being based on filename (e.g., `foo_aix.go`) is clever with all of the hatred I can put into that.
I have shipped software in ~40 different programming languages and distinct dialects over my career. Go does a number of things well, but some of the choices made are frankly bizarre. There are restrictions on how you can name things because of the platform bit and `internal/`, but then there's no recommendation on how to lay out a project otherwise. I’ll still use it any time over most other lower-level languages, but the language itself does not spark joy. The compiler is good and fast, but I have less confidence that I have built the right thing from it than when I use Rust.
Because it’s an excellent fit for some problems, but not so much for others. The ecosystem is definitely smaller and the flower quality compared to say, JavaScript, go, or Java. The performance of the virtual machine is definitely way off the JVM For general computation.
They are slow, have clunky syntax and soundness issues due to lack of strong type system. Now, swap Elixir and F#, and HN crowd, usually signing praises to the former, could stand to gain a lot by using productive language with performant runtime and much better standard library that does not force you into writing code in the exact specific way (especially when it does not fit the use case).
I see so much concern and worry about climate change, how come it's not solved after all this time?
FP, average performance, no static typing, finding people etc ...
Personally I don't like FP.
Structs are arguably better then most type systems, but I don't use it either in my day job and have only used it for toy projects...
Because most developers struggle with functional programming.
which is a shame because I think elixir is probably one of the easiest functional programming languages to learn. most of the libraries use do blocks so you dont' usually think about lambdas unless you're using Enum or List modules and even then, its about as complicated as using reduce or map in javascript. if you know how to use a higher order function, tail recursion and map/reduce/filter, you know enough functional programming to do 99% of day to day elixir work.
has Clojure/Haskell/Ada/Coq(...) took over the World?
is it wrong to love it then?
Because embarrassing little in coding is controlled by these imagineers. There’s no much real difference in programming languages anymore, honestly I’ve never met a good programmer that cared about these, it makes your code fringe and inportable.
Having a language that generates proper errors, does not try to hold your hand incorrectly and instead tells you you are wrong is always better.
Haskell is one of the best in the world in this. But it also happens to be incredibly complex and also weird in it's syntax, to the point it does make code fringe and unreadable.
There is a real difference here - a language that can help you fact check your logic and reason that you are accessing a potentially null value in this context can be better if it's drawbacks don't outweight it.
Elixir isn't static so it doesn't do this but it does blow up a lot compared to other dynamic languages. It's abstraction of proccesses and threads and what not, does actually make your code more portable and more modifyable. It offers a specific paradigm for programming that works great for multiple proccesses and the whole language is built around it. So if you are going to be using that paradigm anyway it would be a better choice.
It's also just pretty how it fits together.
I do not write Elixir/Erlang programmers. But I do think that the actor model and BEAM solve a few issues that I face when writing complex backends, which I do.
For example I have not found a language whose error handling I like - Go and Rust are super meticulous, for example and you get error handling code everywhere, which is good for stability but not great for developer experience. TS/JS/Node suffers from the opposite - error handling is an afterthought and you never know what exception will come at you and from where and bomb you whole server, so you end up relying on cloud solutions running large redundant arrays of processes. Isolation of errors within a process and message passing is a great abstraction.
Another thing is that I get to spend a lot of time on is setting up Redis/Cloud stuff to do basic queue, caching and cron. That stuff takes up time, increases complexity, creates new sources of error and grows institutional knowledge. Using a system which has those built into the language is a major improvement. Vercel/Deno/Bun/etc are solving some of the above by including them in their cloud offerings with relatively good DX, but it still increases complexity, takes you out of code and locks you in with the vendors.
Maybe the programmers you hang around with don't work on large scale message streaming? It's pretty niche