Elixir Livebook now as a desktop app
news.livebook.devThe Elixir ecosystem is growing incredibly well. I am impressed both from an engineering and a product perspective.
It shows that they are playing the long game.
I think they are achieving what I wished happen to the Clojure ecosystem: productive, well designed, respected and popular. Clojure missed the last step, unfortunately.
It seems making a Lisp popular is an impossible task.
That said, I'm not sure if the data agrees with you. I think Clojure is more popular and widely used at this time. Not sure, but I think from what I remember of the few rankings, and just the fact I don't know an equivalent success story to NuBank for Elixir, I think maybe Clojure is at the moment more popular in practice.
But with the amazing learning material Elixir is putting out, maybe it won't last.
In terms of major companies using Elixir, Discord and Whatsapp are built with it. Whatsapp is slowly chipping away at its Elixir for infra homogeneity reasons with the rest of facebook but discord is still all aboard the Elixir train.
WhatsApp uses mostly erlang if I recall correctly.
Neat, I didn't know about Discord. Whatsapp I thought it was all Erlang, are you sure about Elixir being prevalent there?
I wonder if it's only a few teams handling major events at Discord, or if that's truly their default backend service language all their backend teams uses. If so, maybe I should look at their job listings :p
From the discord blog posts it seems that elixir powers the chat system, with rust and python as the other two main languages in their stack.
As for whatsapp, they are mainly a erlang shop and yesterday they open sourced a type checker for erlang:
Since Elixir just compiles into BEAM byte code, like Erlang does, can this type checker be used on Elixir code too?
I think they use Go too.
They moved from Go to Rust.
https://discord.com/blog/why-discord-is-switching-from-go-to...
I would think that NuBank would want to market and promote Clojure to ensure that there would be ample numbers of Clojure developers (since NuBank depends on a lot of Clojure code). Perhaps they do market Clojure in Brazil, but I don't see it elsewhere...
With 213 million people, they do have a ton of devs and a ton of ESL speakers. There’s lots of places like that, like Turkey or SE Asia, where word doesn’t reach NA/wEU.
Do you use pagerduty? They're an elixir shop.
I agree, and I think their move into ML is a genius move that is going to make elixir mainstream. I'm super excited for the future.
I've wondered whether it's easier to add data analyst stuff to Elixir that Python seems to have, or add features to Python that Erlang (and by extension Elixir) provides out of the box.
By what I can see, if you want multiprocessing on Python in an easier way (let's say running async), you have to use something like ray core[0], then if you want multiple machines you need redis(?). Elixir/Erlang supports this out of the box.
Explorer[1] is an interesting approach, where it uses Rust via Rustler (Elixir library to call Rust code) and uses Polars as its dataframe library. I think Rustler needs to be reworked for this usecase, as it can be slow to return data. I made initial improvements which drastically improves encoding (https://github.com/elixir-nx/explorer/pull/282 and https://github.com/elixir-nx/explorer/pull/286, tldr 20+ seconds down to 3).
[0] https://github.com/ray-project/ray [1] https://github.com/elixir-nx/explorer
Definitely easier to add Pandas to Elixir than preemptively scheduled green threads to Python.
Plus sane failure domains: go has had preemptively scheduled green threads from day one but failure domains are really not a thing in go.
Hm, I think I've read somewhere that both Go and Elixir and kinda cooperative. A process in Elixir can yield control after certain number of reductions (function calls I think) and in Go a goroutine can only yield control on function calls, so if you have an infinite loop just adding numbers it will run uninterrupted. Both of them are "less cooperative" than Python with explicit yield statement. Do I get this right? I started digging into concurrency not that long ago.
Yes technically they are not fully preemptive in the sense that an os thread is (the os sends an interrupt which halts the processing at the CPU level), but in both go and elixir the programmer has no control over when the context switching happens, and "function calls" which are the yield boundaries happen all over the place, so it's "effectively preemptive".
Elixir is in practice more preemptive than go (last I checked with go) because you cannot infinitely loop and lock cpu in elixir -- a loop requires you to tail-call in elixir, so that's a yield boundary, and I'm certain that in earlier go that wasn't the case if you `while true {}`
Yes, unless you're in a BIF or NIF every call is a reduction: https://blog.stenmans.org/theBeamBook/#_reductions so if you call out to native code or are using a VM builtin you can lock it up indefinitely but other than that it's not possible.
[I have no experience with Elixir.] Would that be possible to have elixir intelligently manage multiprocessing of python scripts? I would be especially interested in being able to have the scripts talk to each other, but have no idea how it could work, besides perhaps having them communicate by writing to / reading from the same files, which seems risky.
There is actually a case study on elixir-lang.org of exactly this!
https://elixir-lang.org/blog/2021/01/13/orchestrating-comput...
Very informative - thanks!
I think you can have the scripts talk to each other through Elixir via the BEAM/OTP approach: for each script you will have a GenServer module that manages it in a separate process. Call this module MyApp.ScriptServer and put "use GenServer" at the top of the file. This GenServer process will have a client API, which you can write, which will be how you interact with your script. In the BEAM programming model, processes communicate by sending messages to each other's mailbox, so you set up some logic for this and maybe have another module called MyApp.ScriptCoordinationServer that manages some global state.
This is just my amateur guess -- I'd be happy to hear an Elixir/BEAM/OTP expert chime in.
Thanks for the explanation!
This is exactly the solution I was thinking of as well.
> Clojure missed the last step
Oddly, it seems to me that Clojure (the established people in its community) don't really care if anyone is attracted to it.
It seems like the people who come to Clojure do so because they caught a glimpse of it somewhere, did some digging and searching, and decided to give it a try. Not exactly the same as Sun spending $500 million to market Java.
That's not to suggest the Clojure community is unwelcoming or silent, but they generally fly under the radar.
Several years ago I went to a few of the Amsterdam Clojure meetup group meetings, and at that time there were 10 to 20 people there.
Meanwhile, the Elixir meetup would have 20-40! And Elixir was still comparatively young. Of course the Ruby meetup group was even larger, but that's no surprise given the prevalence of Rails in business.
Because of the Java ecosystem, in Clojure you are in much more advantageous position than with Elixir.
Elixir also opens you up to the entire erlang and Beam ecosystems. Not as common as Java, I know, but still decades of knowledge and tooling around it cutting across many industries.
Seems like Beam is a more specialized tool than JVM but if you need that sort of functionality it's phenomenal. With things like built-in performant persistent key value stores and cluster management I'm kind of surprised more people haven't pushed through the "this is useful but lacks broad adoption" barrier.
Well… yes and no. A lot of things built in are pretty spartan and not always appropriate (Mnesia …cough…).
This said it’s the only environment I’m aware of that is built around the idea of cluster, not just a local process that may do some rpc to other processes.
You can find libraries for many things but they are not the same quality and breadth as the big Java ones.
I've heard this, but honestly to me it just seems "specialized for writing and running reliable software".
Seemingly reliable for a specific type of problem, though. Most reliable software doesn't require a distributed network to be reliable, and I don't think binary execution on BEAM is any more inherently reliable than the JVM, right?
The BEAM is designed for a software architecture approach called OTP in the Erlang community -- it is designed around trees of process supervision. So there is an entire robust framework for reliability really built in as a primary concern for the BEAM.
If you need high concurrency, heavy traffic, real-time system with high resilience, there’s probably nothing out there that comes even close to BEAM. It is definitely possible to write such a system in Java or Rust, but most likely at huge cost for something that you can get out of the box elsewhere. BEAM has narrower scope, but in that scope it’s probably far ahead of anything else.
I tend think the opposite. Developing for Android I found the Java build process and maintaining a solid development environment a headache. If that follows on to Clojure I can see why it wouldn’t be as popular even if the language itself is overall better.
My experience would make me agree with you. Developer experience around tooling in Clojure makes a lot of assumption around prior knowledge of Java ecosystem.
Elixir, on the other hand, welcomes the newcomer better.
Except using Java ecosystem as-is would not be idiomatic in Clojure, which means you need an ecosystem of wrapper libraries. I recall someone making similar argument on some other post, on how it adds more space for bugs, performance issues, and still making it hard to work with. The context was in comparison to Kotlin, where using Java ecosystem is much nicer experience.
I see it the other way around -- because of the BEAM ecosystem and programming model you are in a much more advantageous position than with Clojure.
No Linux download :-(
Nevertheless it's fairly easy to install[1]. If you have docker, you can just run the image to try it out:
docker run -p 8080:8080 -p 8081:8081 --pull always livebook/livebook
If you want a "native" install, after you have elixir installed you can use
mix escript.install hex livebook
livebook server
> No Linux download :-(
They discussed it, but no one has taken it on yet. I think Wojtek Mach(the desktop initial author) is a mac user, and they actively wanted to provide a way for windows users to easily try it.
We don't plan to add Linux support ourselves at the moment but we'd love if someone contributed that. As was mentioned elsewhere it might be challenging to provide a bundle that works across different distributions so investigating the possible options would be an important first step.
That's what I assumed was the case. Seems like a good opportunity for the community to take on for major distros.
An AppImage or Flatpak package would make a great starting point that works across distributions. Making Livebook more accessible to Linux developers would improve its adoption, as well as the adoption of Elixir, Phoenix, and Fly.io.
Have you looked into nix? Works for mac and linux.
Props to the Elixir community, it seems for some reason they've attracted a lot of people who have the will, the free time and the talent to provide amazing educational content. I've never seen a language of this size have so much quality learning material and tooling.
I'm currently reading an Elixir Patterns book (https://elixirpatterns.dev) which releases a side-by-side Livebook for the material with each chapter.
It's a fantastic way to learn and a huge advantage over other languages, I think, to help build up the Elixir community.
Co-author of Elixir Patterns here :).
Really appreciate the kind words and happy to hear that you are enjoying the book! Hugo Baraúna and I have really enjoyed putting this book together and feel that Livebook has become pivotal in explaining certain topics and really understanding how things work under the hood. I wrote a blog post about some of my thoughts around Livebook as a learning tool on the DashBit blog: https://dashbit.co/blog/livebook-as-an-educational-tool
I love the concept of local Web servers as GUI substitutes, and hope we get many more of those. Tachidesk is my favorite example. Thanks to Phoenix Liveview, Elixir is becoming a great option in this space. This method gives us every benefit of Electron and almost none of the downsides.
I have the opposite opinion. These things tend to get lost in my tabs.
Pin your tab.. But I hear you, having the same problem, there are trade-offs for sure.
Or just bookmark them. They usually have the program name in the page title, on firefox just begin with * on the url bar, or press B if you have Vimium.
This approach has been around a lot longer than Electron too. Privoxy, i2p, CUPS, etc. were/are all local-first software with Web-based interfaces.
I feel the same. Wonder if the author can provide a bare-bones repository/template with Elixir and WxWidgets to create similar apps.
This is awesome! Starting it up from the terminal and going to the url was not super convenient for quick tasks and scripts, this is much better.
Edit: one thing that would make it even nicer is if it would remember the folder and/or recently opened notebooks.
Recently opened notebooks are on the roadmap. ;)
Amazing! Can't wait.
Why would you use a notebook for quick scripts though
I often need to inspect/transform some JSON/CSV/text/whatever data, move around/organize a bunch files, and similar things. I find that notebooks work great for that as you don't need to set up an environment, you can easily collect and rerun your scripts if you do need them again, and the output options are great for debugging and looking at data.
I know this is a Livebook love-fest, but you may also consider setting up org-mode sometime. There's a lot to love about org-mode but to stay on subject, you can write your documents in a plain text syntax akin to markdown and any of the code blocks that you add in as examples are _executable_. And you can take the output from one and pipe to another.
At work, for example, we have to hit a remote API and the secrets are stored in AWS secrets manager. I have some documentation that has a block describing how to get the secrets and some other documentation elsewhere describing how to issue a post request and I can pipe the output from one block into another and receive the output either in the same file or put it somewhere else.
The one downside with this extreme flexibility is I sometimes catch myself squirreling away useful functionality in my own notes instead of adding it in a utility CLI or similar where it can be shared with the team.
If having access to a scripting language is duct tape, piping output from one script to another in this way in a single file is like having inter-dimensional duct tape.
Visidata† and Datasette†† are both excellent for rapid exploration. Both are available via Python pip.
This idea is incredible, and extensible beyond these notebooks. I’d like to see something like this as a general code authoring/collaboration tool, would be nice to see when I’m working in the same code as a co-worker, or supercharged pair programming. Obviously scaled too large this would be chaos, but on a 6 person team I think this would be killer.
A super cool notebook, you should give it a try even without knowing Elixir with the tutorials coming with it :D
I’d be interested to see the details of how they did that. Very cool!
You can have a look at the code: https://GitHub.com/livebook-dev/livebook/tree/main/app_bundl...
I, too, am interested but I'm not terribly literate in elixir yet. I am aware that you can compile executables in elixir with a release command but I'd be curious to hear more about (or at least be pointed to the specific spots in the code) how the bundling was done to give you a discrete application per platform, with app icon and everything.
Having a script that opens a web browser pages is simple enough, but the extra sugar on top to wrap in an app bundle that shows up in your task bar or dock is something I am very curious about.
EDIT: Someone further down the thread linked to a more specific place and that led me to the macos.ex file[1] (which is what I was most interested in, being on macOS myself.
It looks like at least in the case of macOS it will build the bundle by hand, template in enough swift code to build the application, sign and notarize a bundle, and add the compiled elixir program in there. Pretty interesting. I guess the idea is that you write just enough swift to manage the life cycle of the compiled elixir application. It's an interesting approach but it seems like it would be frustrating and fragile.
[1]: https://github.com/livebook-dev/livebook/blob/355527f7e8034e...
Right, so our goal was to allow configuring the bundle with app icons etc and importantly custom URL schemes and file types [1]. And then macOS- [2] and Windows- [3] specific launchers capture the "open file" and "open url" events and trigger the Elixir side of things. Another reason for native launchers is if for any reason we couldn't start the Elixir side, we want to let the user know that something went wrong and point to logs. (We have that on Mac and will have on Windows very soon.)
[1]: https://github.com/livebook-dev/livebook/blob/920f70817efbac...
[2]: https://github.com/livebook-dev/livebook/blob/920f70817efbac...
[3]: https://github.com/livebook-dev/livebook/blob/920f70817efbac...
I know linux is a future supported platform but as far as I'm aware, there's not a clear unified way to build bundles in this way across linux distributions. What kind of things are you folks considering if you don't mind my curiosity.
We don't plan to add Linux support ourselves at the moment but we'd love a contribution.
are there any plans to also have LiveBook open up in a separate wrapper app, like Electron?
It would be crazy awesome if someone could figure out how to layer python underneath this. There is a very nice feature set in livebook that doesn't really exist in jupyter.
As others pointed out, the app uses wxWidgets for the UI and Elixir for backend, along with Phoenix. I think similar apps can be created using wxWidgets with Python by replacing Phoenix with another framework, maybe Flask or Django.
Yeah you could but it hasn't been done. I've struggled with jupyter/ipython for a decade. CRDTs like automerge and OTs exist, but they're not in jupyter.
It's simply put, not that easy to manage distributed live state in a sane fashion with python (see openstack), the programming model of the underlying language puts obstacles in your way. Meanwhile Erlang was created by a physicist who understood that the speed of light is a thing and consistency of data across distances is a real issue.
Any tips why would you want to learn Elixir or what would you use it for?
I tried to learn it a couple of years ago. Bought a book and couldn't shake the impression that it was created to be complex and hard to reason about or maybe I am wrong and it is "easy", but has a steep learning curve?
That being said, I looked at Go around the same time and I clicked instantly. Easy to read, simple and yet powerful.
It shines at creating highly concurrent applications with a great deal of built in features to provide reliability and resilience. Sounds a lot like a server, and that's what much of the modern use for Elixir is. In particular, one area where it shines is stateful, long-lived requests (e.g, websockets) where many other concurrency models will either rely much more on databases to track state or potentially get choked up.
If nothing else it's neat to learn as one of the few somewhat well used languages in modern times that actually adheres to Alan Kay's original conception of OOP.
Was excited for this. However on my M1 mac, it launched once, and now it won't open. Tried killing all processes and now nothing.
Sorry about that! A bug report at https://github.com/livebook-dev/livebook/issues/new would be very appreciated, especially with relevant parts from ~/Library/Logs/Livebook.log.
Thanks for the log location - I killed the erlang process and now it works - it launched in a browser window with no app icon though?
Yeah there's no app icon in the Dock because we don't have any GUI besides the menu bar icon.
As someone who also doesn't know Jupyter, what sort of fun and useful things would I do with this? It looks like I can add content and some code blocks that I can execute. I guess taking notes and tinkering?
I'm messing with Elixir just for fun, so this caught my eye.
not sure what the livebook is--the landing page is not sufficiently clear or coherent.
> not sure what the livebook is--the landing page is not sufficiently clear or coherent.
The difference between features and benefits. It's good at telling you the features (what it does) but not so much the benefits (why you would use it, how it can help you).
A common issue, but more understandable in this case than most as your average visitor to the Livebook site will likely know they have a need for it already, rather than end up there based on a 'random' link (like this thread).
The headline on the landing page is "Write interactive & collaborative code notebooks in Elixir".
That's pretty clear and coherent to me.
why would i need to do that instead of Jupyter Notebooks?
Because you want to write Elixir instead of Python/R?
I can already do that with IElixir: https://github.com/pprzetacznik/IElixir
> Write interactive & collaborative code notebooks in Elixir
Think Jupyter Notebooks (if you're familiar with the python world), but collaborative and for Elixir.
I think this is a fair analogy, but maybe I'd add that LiveBooks uses MarkDown markupfor the non-code part of the notebook, make it far more readable, and with more visualizations coming in.
For example, recently an addition to add automatically generated sequence diagrams to Livebook Kino (https://github.com/livebook-dev/kino/pull/165). So generate code which shows a supervision tree, and automatically create a visual diagram of the code in one place. Pretty great for learning and explanatory material.
if they had written this in the landing page, i would not have written my rantish comment.
You're correct that it's not really captured in the prose of the article, but the first image does inform you.
Related question: would people pay for this kind of localhost desktop app? I'd love to sell an elixir app without bothering Electron or Rust based engine.
Is it by any chance an Electron app?
It uses an erlang binding for wxWidgets[1].
What for? The Livebook interface is built with Phoenix Liveview.
The window it runs in.
Is this required in some OS and not another? A comment in this thread claims it just uses a browser tab.
It's used for the desktop app, as you can see here[1].
[1] https://github.com/livebook-dev/livebook/blob/355527f7e8034e...
Thanks for linking this. I was interested in how it worked but I didn't know enough elixir to get around the codebase by myself. This entry point is very informative for how they managed to do it.
I just looked at the macOS part, but the tl;dr for that is they seem to have compiled the elixir program into a binary and wrote enough swift to manage the lifecycle of the elixir program and signed it and notarized it and prepared it for shipment using CLI via shell! invocations. I didn't look at the windows way of doing things but I imagine it's similar.
No problem! The pr here[1] is also instructive, and here is the one for windows[2]. It seems similar-ish.
[1] https://github.com/livebook-dev/livebook/pull/865/files
[2] https://github.com/livebook-dev/livebook/pull/1032/files
Since Livebook is a web app, I would assume so.
I just installed it and it actually doesn't use Electron. It's basically just a local web server that runs on your computer, and opens a browser tab in your default browser pointing to http://localhost:60904/
On macOS this is a great app for turning browser web apps like these into dedicated 'apps':
I use it with YNAB
Or Fluid which is free except for a few features which cost $5 (https://fluidapp.com/)
How is Livebook using iframes? Is each notebook cell an iframe?
No support for Ubuntu?
Not as a downloadable desktop app, yet. You can run it in docker[1].
How many layers can we do!! I’m guessing elixir to JavaScript to v8 or whatever the js vm was called in chrome. Then jit then binary.
Communicating over a network or one of those chrome sockets to an elixir or erlang vm so someone else can manage your data.
Life is good
Not quite, this is just your standard server side rendering for providing a convenient GUI alongside your erlang vm. There's no layering here, just bundling of things that are usually distributed separately (editor and vm).
And the interaction is handled by Phoenix LiveView (html pushed trough websockets)