Erlang/OTP 24 highlights
blog.erlang.org> there are still 260k lines of code added and 320k lines removed
So a net reduction of 60K lines of code? And yet functionality was added to the system as well. That's praiseworthy IMO.
Imagine if "we did more with less" infected much of software development today.
Recently I read somewhere that writing long prose is easy but writing something succinct takes way longer. I think that translates to writing code as well.
“ I only made this letter longer because I had not the leisure to make it shorter.” Blaise Pascal
Ironically that's quite a pithy statement.
Brevity is the soul of wit.
Laziness is the father of efficiency.
Imagine if we had a metric of "how easy is the code to understand and work with" rather than "can we do it with less lines of code".
I think that's where people swing back to DAMP, or at least the rule of 3.
I'm not trying to fractally compress my code. I'm trying to make it succinct. Those two things are very different. Now if I could only convince a particular coworker of that...
I think every good programmer has these rules of thumb, from most important to less important:
1. Make it so you'll understand it in 2 minutes after being away for a year. Positive side effect: it's very readable for your colleagues.
2. Make it brief.
3. Make it fast.
4. Make it pretty (although often times readable + brief code is pretty but obviously it's subjective).
I've gotten to a place where I start with a sketch of how I think my code ought to be used and then work backward into the implementation.
So far I've gotten a lot of pushback from others on my team. And, what starts as "oh, you should just have to ____" gets really messy as the implementation takes shape.
So test-driven development (where you write the user -- the test -- first, and then the implementation)?
I think so, yes. But I start "before" the test? I write a use-case. Almost a user story, but in code. Sometimes I add comments inline with the code to try and sort out what abstractions are useful.
Still TDD, this could fall under the use of Cucumber where the integration test DSL reads almost like English.
So kudos to you for using good practices.
Those two are not exclusive.
> So a net reduction of 60K lines of code? And yet functionality was added to the system as well. That's praiseworthy IMO.
Yup.
> Imagine if "we did more with less" infected much of software development today.
Taken to its logical conclusion this should mean that the ultimate piece of software, that does everything and does it perfectly, is 0 LoC.
Hmm, fells like I need to check my logic here...
OMG, YES! The JIT is here... and holy crap, if you haven't tried it yet... it's awesome (everything feels snappier). I'm particularly stoked for the receive optimizations and process aliases.
BEAM just gets better and better. It's a good time to be an Erlanger/Elixirist...
I just ran programs from computer language benchmarks with OTP24. Every program ran considerably faster almost 30% speed up.
I was hoping our `mix test` would be faster, but it doesn't appear to be. It would be nice if this got some attention fro the Elixir team.
`mix test` _is_ fast for me... I've only seen slow tests when folks are misusing timeouts (generally speaking); what problem are you seeing?
The delay is all in compiling exs files. It uses Kernel.ParallelCompiler to compile every .exs file, so it's very CPU/core dependent. On my weaker laptop, `mix test` takes nearly 10 seconds to just start.
I've looked into this in more details in the past. We've had success just writing our own test runner and avoiding exs files. But re-implementing things like running tests based on line number, or integrating with external tools (like excoveralls) has been a dealbreaker.
FWIW, I recently pushed a commit to master that made loading of Elixir's test suite 33% faster (from 15s to 10s): https://github.com/elixir-lang/elixir/commit/2eb03e4a314c0e6...
Unfortunately, it is a bit too large (and too late) for v1.12, but if loading times have been problematic for you, it would be awesome if you could try master out and let us know in the issues tracker (or in the commit) if you see any improvements.
Can we expect it for 1.13?
Yes.
What made your test runner faster? We would be very interested in porting those optimizations to ex_unit.
Are you sure that you're not including application startup in that 10 second measurement? I've seen code bases where horde or some other clustering was enabled in the test env and causing a ~5 second delay on application startup.
Really? I'm surprised that is your test bottleneck. I've mostly seen it be actually slow tests and things that can't be asynced.
It's noteworthy that the JIT doesn't (yet?) do runtime optimization or specialization, so gains should be moderate. The very low end of double digits.
Not comparable to going from a Javascript interpreter to v8.
But it's a great starting point.
From my recollection (interviewed OTP team on this stuff once) they don't really intend to go there either. They are a very small team comparatively and maintaining that kind of runtime optimization would likely be unwieldy. (https://devchat.tv/elixir-mix/emx-114-just-in-time-for-otp-2...)
It won't be anything like V8, entirely correct, but it brings some VM code to native performance, beating NIFs in some cases.
I think some RabbitMQ tests reported 30% increase in throughput which is pretty wild.
Seems like WhatsApp / Facebook would benefit by contributing some resources towards this...
Arguably, yes :)
Erlang has long, long needed better error messages. These changes look very welcome.
Overall this looks like quite a nice set of improvements. Kudos to the team.
The column number for errors and warnings is a welcome, long overdue addition.
Working with Elixir on a daily basis is mostly a pleasure, but those "ArgumentError" errors have been super annoying.
Looks like Elixir 1.12 will take full advantage of EEP 54:
https://github.com/elixir-lang/elixir/releases/tag/v1.12.0-r...
Awesome, thanks for linking that! I was wondering how these improvements would flow through to Elixir.
Looks like we get the JIT performance improvements for free.
And unrelated to the Erlang/OTP changes, Elixir 1.12 looks awesome. Totally small, but such an unexpected little quality of life improvement, Kernel.then/2 for pipelines, looks great. I love the core team's focus on the UX of the language.
Totally agree.
I have just a few remaining complaints about the language at this point, and Kernel.tap/2 and Kernel.then/2 will solve two of them.
When Jose mentioned a few years back that Elixir the language was more or less "done" or at least stable, and that they would focus on ergonomics and UX going forward, I remember getting a little worried. But I’ve found myself agreeing more and more - there’s not much I miss in the language itself, and projects like Nx, LiveView and LiveBook have shown that it’s an excellent foundation to build very powerful and modern stuff on top of.
As someone who has been programming with Elixir for my day job for the past few years, I find this aspect of the language to be super pragmatic and productive. It's a nice feeling to not have to chase new language features and syntax and focus more on the problem at hand. In addition, I've never felt limited by the language given that the underlying constructs are so powerful (message passing, immutability, pattern matching, etc). Glad that Jose made the decision that he did.
Curious, do you use typespecs and dialyzer? If so, how do you find it?
Elixir checks pretty much every box I'd want in a language, but after dealing with nil in Ruby for years and having fun with TypeScript... I'm feeling more drawn to working with type systems.
I find working with nil in Elixir to be quite reasonable. One way in which elixir is different is that you have different operators for when it must be boolean, and nil is not tolerated for those operators (and vs &&). This coupled with the convention of using ? at the end of functions which emit boolean makes things easier.
The one thing I wish is that people stopped writing boolean functions with is_ in front (that is supposed to be only for guards, but not everyone follows that convention).
If you want type systems, you can probably work with Gleam, in the future, I imagine there could be great interop where you can just drop in a .gleam file in your elixir code base with zero hassle attached, and have parts of your code base that are completely type safe, and let Elixir handle all the risks of IO and other effects.
But, this might just be my wishful thinking.
For what it's worth, I spoke last year about using Gleam to develop a type-safe core for a LiveView application: https://www.youtube.com/watch?v=UCIcJBM_YDw
I think that what you're looking for is already possible today, and that things only get easier over time.
I agree it's possible. The pertinent question is: how easy and ergonomic it is to do? Friction kills good innovation very often.
I've kept an eye on Gleam. Essentially all of the programming I do is based around the web so I'm kind hoping a web framework shows up at some point.
I'm not a huge fan of Dialyzer myself. I would put it strongly in the "much better than nothing" category rather than the "usable and useful type system" category. I always write specs for my functions and types, and while they sometimes catch bugs, they're not quite as expressive as I would like them to be.
I suppose you could write Elixir in a style that was more type safe by writing in a less polymorphic or recursive style, but the language does not lend itself well to it. Structs and maps are mostly fine, discriminated unions less so.
Answering from similar experience. I personally use them both, dialyzer as part of the Elixir Language Server and typespecs when I feel something needs more clarity and definition.
Depending on what itches your types scratch for you it might be enough, might not. I've never wanted more type system in my Elixir personally.
I generally verify types only at the boundaries of my application (or very critical modules) using norm[1].
Either you have a strict type system that does not have an "any" type (yes, I'm looking at you Typescript), or you have a flexible type system like Python/Erlang/Elixir and you do runtime type checking whenever it's needed.
I'm writing more Typescript code than I would in Javascript for almost no type safety benefits (but for documentation, it's awesome).
Dialyzer and typespecs are better than nothing but not with much because they can introduce a lot of friction along the way, for not much benefit.
As another poster said, it can catch the very occasional potential bug but to me at least it's rarely worth the hassle.
I miss static typing after I got back to Elixir from Rust but the BEAM will likely never be statically typed.
To that end, I found utilizing metaprogramming to generate normal and property tests to be much more productive use of my time, with a measurable impact to boot.
I'm in the same boat. Love Elixir, but I always found the typespecs to be a huge pain (and slow!) vs. type systems like typescript.
I'd love to see experiments to replace typespecs with a more ergonomic type system.
"When Jose mentioned a few years back that Elixir the language was more or less "done" or at least stable, and that they would focus on ergonomics and UX going forward"
That's awesome to hear. I think there's a lot of language communities in the 10+ year range (and look, that's right where Elixir is) could stand to have that recognition. I'm getting kind of saddened by the number of languages I see that are good languages that proceed to festoon themselves with so many new features that they become quite difficult to use.
Seems like there is less and less reason for not using Erlang for distributed coordinated compute broker tasks.
I have a stateless java backend servicing REST APIs. These backends are load-balanced by nginx.
Now the time is approaching where I need to introduce some form of global state (that involves global caching, message passing, registering/discovery of known workers, periodic (cron-like tasks), etc).
I would much prefer a single tool to add to my backend stack (currently Java + postgres) to cover most of the above needs.
With the performance improvement + persistent_term [1] -- in current Erlang, I basically get:
- a light weight distributed K/V cache
- a ZeroMQ like on-wire messaging system (built into erlang)
- discovery/coordination
- a distributed compute grid
- a way to write my own routines within that compute grid, and have them exposed via Java interface to my existing backend.
I do not need 'fastest' possible performance or least memory consumption. I just need them to be 'reasonable', 'known' and 'controllable' (not to exceed some baseline).
Erlang is just looking better and better (and I prefer its syntax to Elixir, for some reason.).
Do consider Elixir for its metaprogramming (macros) and the goodies that come with them, namely fantastic libraries and frameworks like Ecto and Phoenix.
Nice update on the error message!
Speaking of error message. I love that Erlang gives the exact parameter values in the call stack. (I guess this is generally possible because immutable data structures are enforced in the whole language?) It saves a lot of time than just speculating with call stack only.
I love the new error messages. Rust seems to have started that trend, Python also added better errors recently.
Okay, lots of people arguing about this, so I'm gonna reply to you, the top-most person in this sub-thread :) I think the reason that it's easy to argue is that you can be talking about similar but slightly different things.
When Rust was created doesn't really matter, exactly. For a very long time, errors looked something like this:
At some point, "improving the error messages" became a project goal, and Jonathan Turner decided to take this on. This is described in https://blog.rust-lang.org/2016/08/10/Shape-of-errors-to-com... . The post explicitly cites Elm as inspiration because Rust was directly inspired by Elm in this regard.hello.rs:2:4: 2:16 error: unresolved name: print_with_unicorns hello.rs:2 print_with_unicorns("hello?"); ^~~~~~~~~~~~~~~~~~~Yes, other languages may have started this trend earlier. Yes, maybe they influenced Elm, which influenced Rust. Yes, Rust may be "older" than Elm. But if you ask the people who began this effort, they will name Elm as the inspiration.
(And yeah, then you can try to argue about who has influenced the broader public, which is effectively impossible to prove, IMHO.)
Today that error looks like
incidentally, and there's also JSON output, and if you're not on a terminal, you get the line numbers like before... lots of things are improved. This particular error doesn't show off some of the nicer things. Esteban Küber has taken up where Jonathan left off, and has been doing amazing work.error[E0425]: cannot find function `print_with_unicorns` in this scope --> src/main.rs:2:5 | 2 | print_with_unicorns("hello?"); | ^^^^^^^^^^^^^^^^^^^ not found in this scopeexpressive diagnostics was one of clang's early selling points: https://clang.llvm.org/diagnostics.html
Clang was released 2007 and was usable 2009/2010-ish. Rust dev started 2010.
I'm not saying the trend of having good diagnostics was started by clang, but it's a more believable than the claim that it was started by Rust.
--
Rust-the-language is nice, but the Rust community feeling the need to mention Rust on every unrelated thread is a bit of a turn-off for me.
Template errors with g++ vs clang++ are what made me abandon the gcc toolchain for my dev environment (I still build against both compilers in my CI/CD though).
Rust wasn't even a thing (not as hype and mature) at that time.
Rust is nice, but the hype train is toxic (and that's true for every language/technology).
GCC also copied clang's excessive diagnostics and I'm seriously thinking of making a fork just to strip them out. I doubt the developers would accept a patch that implements --stfu.
Every change breaks someone's workflow. But you are the closest I've seen to the person complaining here - https://xkcd.com/1172/
Yeah. It's funny how back in the day people used to make fun of excessively long compiler errors involving templates.
Now everyone seems to praise the compiler barfing three screenfuls of text and code and explaining the include hierarchy of my project and expanding all the macros and making bogus suggestions because I mistyped a variable name.
It's gotten to the point where locating the actual error message is literally more work than fixing the code. It doesn't help that this interferes with navigation in emacs and trying to jump to next error instead takes you to the next #include line gcc wants you to learn about.
Does -fdiagnostics-plain-output not do what you want?
$ cc -fdiagnostics-plain-output c.c cc: error: unrecognized command-line option ‘-fdiagnostics-plain-output’It's new in gcc 11.1
> Rust seems to have started that trend,
No it didn't. Lets not try to rewrite history here, clang started the trend of meaningful error messages, gcc quickly caught up.
Rust was inspired by Elm, if I’m not much mistaken.
People around here give Elm way too much credit for inventing things it didn't invent or wasn't even the first to ship useable versions of, but I think its fair to give Elm full credit for elevating the quality of error messages one can expect from a language or framework.
Rust was started in 2010, and Elm in 2012.
https://en.wikipedia.org/wiki/Rust_(programming_language)
https://en.wikipedia.org/wiki/Elm_(programming_language)
That said, the concepts in question are all much older. If you're ever bored, check out the "Influenced by" sections of the Wikipedia pages on programming languages. It's amazing how old so many of the "new" ideas really are.
> Rust was started in 2010, and Elm in 2012.
That doesn’t preclude learning from younger langages in any way.
The trend of providing really helpful and valuable error messages (especially compilation) really started with Evan’s “compilers as assistants” and “compiler errors for humans” from 2015, although there had been forays into improvements to e.g. error localisation from clang.
And Rust’s improvements absolutely come from there, as acknowledged by Jonathan Turner’s 2016 “shape of errors to come”: https://blog.rust-lang.org/2016/08/10/Shape-of-errors-to-com... as well as his “new error format” proposal / rfc which eventually lead to the change: https://github.com/jonathandturner/rust_proposals/blob/maste...
I remember Ada always having very precise and helpful error messages like this one:
literal_string.adb:5:33: warning: wrong length for array of subtype of "Standard.String" defined at line 5 literal_string.adb:5:33: warning: "Constraint_Error" will be raised at run time
then when you run the app it behaves as advertised
$ ./test
raised CONSTRAINT_ERROR : literal_string.adb:5 length check failed
There are many others and when I saw llvm improving error messages for C and C++ (which I saw happening before Rust was a thing) I always thought it was inspired by Ada and it's helpful messages like:
expected private type "<type name>" defined at ...; found type "<type name>" defined at ...
"<name>" is undefined (more references follow); possible misspelling of "<name>"
and the many others that were just there when I first tried Ada around 2008/2009.
clang also played that game around that time (these are the two I had in mind when thinking of this)
True, Evan actually mentions Clang (noting that he’d met people who’d switched from gcc to clang due to the error messages) in “compiler errors for humans”.
I love it how good ideas just spread everywhere in open source. Everyone's life improves and usually people aren't obnoxious about it, which tends to happen in more politically charged topics (i.e. when there is a company pushing a narrative such as google,apple or microsoft)
Improved error messages is great. But one area specifically that is in desperate need of some love is the type errors from Dialyzer. They are rarely helpful to identify what is actually wrong in the code. Typically the error message will point to something several layers up or down in the call stack and prints out a huge blob of unreadable and unhelpful type signatures. Most of the time the best you can do is to simply compile and run the system to figure out what is wrong with the types. The obscurity of the type errors remind me of C++ template errors.
Part of the problem here is that dialyzer is a quite ... spaghetti codebase, that noone really want to fund that work and that noone really knows how this stuff work that is interested in doing that work.
There is a lot of research work to do and noone to pay for it. That said, if you know of a company interested to fund this work, i am interested and would love to hear more about it.
I've been looking forward to the JIT for months. This is fantastic!
Years here
Slightly off-topic from someone who hasn't touched Erlang in almost 20 years: if my app is distributed with Erlang/Elixir, I kind of feel like my choice of database should be something with excellent horizontal scaling too. Can someone offer some insight into the trends in 3rd-party tools like databases for Erlang applications?
Erlang already comes with a distributed DBMS called Mnesia https://erlang.org/doc/man/mnesia.html , which under the hood uses ETS/DETS depending on the configuration. In distributed Erlang projects you'll find that or abstractions over it. Mne sia makes the most sense in mu opinion when you only have 1-2 relations or just need some kind of distributed cache.
In Elixir apps you'll frequently find the aforementioned Ecto "ORM", which has adapters to different DBMSs like MySQL, PostgreSQL, and even Mnesia.
The one sort-of thorny piece with Mnesia is querying the DB. The syntax for select is not super straightforward and involves writing matchspecs (http://erlang.org/doc/man/ets.html#match-specifications) which are non-trivial.
Mnesia lets you do a lot of cool things (in memory DB by default! Super fast!) but it also has some pitfalls (Doesn't write to disk by default! Lose all your data when you restart the BEAM!).
Generally, I think it's a fine system if you're willing to put in the time to optimize it.
Postgres? ;) In all seriousness though, I wouldn't really expect database needs for Erlang & Friends to be that different from other languages. My current employer has a vertically scaled AWS RDS Postgres as companion for an Elixir app and it's worked great. (Ecto in particular is an excellent ORM.) If your app truly needs zero downtime or ultra-low latency then there are other options.
I've had interesting exchanges with a guy who does work on CouchDB. That is built with Erlang and should scale quite well in what sounds like a resilient way. Haven't dug in.
Horizontal scaling of the DB seems like a problem in many architectures. For SQL I'd be looking at CockroachDB as it is Postgres-compatible and is built with scaling out in mind. Haven't tried it though. Most of my work hasn't needed that for the DB recently.
I love the notion of co-located processing and data, but most of the data stores dont quite embrace it. Well actually Riak does quite well if you're fine with a dynamo like kv store. The riak library is actually pretty nifty for distributed computations. But with Basho out of business it's harder to justify. Plus KV stores really aren't as usable as good ole sql. In that realm there's ActorDB, but I've not been able to figure out how to run it inside my own beam cluster rather than standalone.
Hadn't heard of ActorDB. Interesting. But is it alive? Seems inactive.
It seems dead and it relies on some (older) rebar stuff. And the custom "actor" syntax would probably break Ecto..
Probably CockroachDB (https://www.cockroachlabs.com/)? It's wire-compatible with postgres so any library with a postgres driver will work.
Elixir has the excellent Ecto package which talks to many SQL dbs, Postgres being the default.
BeamASM vs HiPE?
Since BeamASM doesn't support HiPE - has anyone seen benchmarks of BeamASM (JIT) vs HiPE. I've searched and searched and can't find such analysis.
(Super excited the JIT work is seeing light after 10+ years)
fwiw elapsed seconds
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...Erlang/OTP 23 [erts-11.1] [hipe] Erlang/OTP 24 [erts-12.0] [jit] binarytrees,hipe,1,7.531 binarytrees,erlang,1,11.768 binarytrees,hipe,2,4.172 binarytrees,erlang,2,5.149 fannkuchredux,hipe,1,59.079 fannkuchredux,erlang,1,73.151 fasta,hipe,1,57.006 fasta,erlang,1,50.843 fasta,erlang,2,20.209 knucleotide,hipe,1,92.662 knucleotide,erlang,1,80.360 knucleotide,hipe,3,86.793 knucleotide,erlang,3,70.949 mandelbrot,hipe,1,118.623 mandelbrot,erlang,1,48.756 mandelbrot,hipe,2,101.211 mandelbrot,erlang,2,46.154 mandelbrot,hipe,3,87.793 mandelbrot,erlang,3,44.633 mandelbrot,hipe,4,84.878 mandelbrot,erlang,4,44.976 nbody,hipe,3,140.025 nbody,erlang,3,100.630 pidigits,hipe,1,8.791 pidigits,erlang,1,8.008 pidigits,hipe,2,8.515 pidigits,erlang,2,8.673 pidigits,hipe,3,7.935 pidigits,erlang,3,7.748 regexredux,hipe,6,42.757 regexredux,erlang,40.402 revcomp,hipe,1,25.622 revcomp,erlang,1,23.070 revcomp,hipe,3,188.990 revcomp,erlang,3,155.024 revcomp,hipe,4,122.507 revcomp,erlang,4,106.456 spectralnorm,hipe,1,92.347 spectralnorm,erlang,1,62.519 spectralnorm,hipe,2,11.176 spectralnorm,erlang,2,11.460HiPE support is getting wonky, I don't know if many people use it, and I'm not sure it supports the latest OTP versions.
The Erlang folks are looking for maintainer volunteers to continue working on HiPE support, they don't have the manpower right now to maintain it themselves.
I'm not sure if there's a point in the history where you can run with (this) JIT or with HiPE on the same commit. Which makes an apples to apples comparison difficult. If you compare HiPE on OTP 23 with JIT on OTP 24, you're also getting the large amount of other changes as well.
Both HiPE and this JIT have drastically different improvements depending on the specific code that's running, which makes it challenging to have a real world benchmark as well.
There's a comment here: https://blog.erlang.org/the-road-to-the-jit/#maturing-the-ne...
I feel like I saw graphs in one of the presentations, but I don't recall which, or if it was about the final iteration of the JIT.
I suspect HiPE would beat the JIT on tight loops, but the JIT wins in general because of the lack of switching cost between native and interpreted code.
I believe HiPE is going away in the future.
Always puts a smile on my face when I read these, the team does an amazing job. Cheers! :)
Sometimes I'll sit there, look at the error trying to decipher it, and I forget who I am and what I was doing.
Great work on the improvements!
I'm going to miss those `badarg` errors :D
I know you're being sarcastic but I will definitely not miss those when working with Elixir......
Seriously. I'm forever forgetting that Erlang functions called from Elixir often accept charlists as args.
Found the person who likes writing an in-house ETS wrapper on every new job!
One past thread:
Erlang/OTP 24 Release Candidate 1 - https://news.ycombinator.com/item?id=26260127 - Feb 2021 (15 comments)
Adoption of JIT on robust platform like BEAM is excellent news. Although i would wait for one or more release major releases till I see the significant improvement in computational aspect of the code.
bring on the ARM support