iroh

Running iroh on embedded systems

Iroh works very well on modern hardware of various sizes, from smartphones to big multicore servers. But what if you want to use it in an embedded context?

I have a little hobby project for home automation. Naturally I thought about putting iroh on it.

Of course it would be easy to just get a Raspberry Pi and put iroh on it. But that would double the cost of the project and also not be very elegant.

So let's instead try to get iroh to work on some really cheap embedded device, an ESP32. More specifically an Espressif ESP32 WROVER chip that is the part of many cheap ESP32 dev kits.

You might think that it is impossible to get a complete QUIC stack including TLS to run on such a small device. But on the other hand, this CPU is about as powerful as the first 32 bit computer I owned, an AMD 386DX40 with 4 MiB of RAM. Back in 1992 this was considered a very powerful computer. It even ran DOOM.

Surely it should be possible to run iroh on that...

Of course the ESP32 is a very limited environment compared to even the smallest Raspberry Pi. You get around 4MiB of flash for the application binary and ~500 KiB of internal RAM. For some variants, like the one we are working with, you get some additional memory.

So it is going to be a tight fit.

Getting started

Using rust on an ESP32 is very well documented. There is an entire book for it.

We are not going to spend too much time with the main point of an ESP32, input and output via GPIO ports. We just want to set up a tiny hello world project and then turn it into an iroh hello world project.

To set up a simple hello world project, there is a project template.

This will give you a rust project that uses a proper operating system (FreeRTOS). This is required since iroh needs std and TCP/IP and WiFi support.

We now have a minimal hello world project with the ESP32 tool chain set up, and can run it.

Building this for the first time will download a custom toolchain. Some variants of the ESP32 use a RISC-V architecture, which does not require a custom toolchain.

Running it will try to flash it on a connected device, so you need an ESP32 connected to your development machine via USB-C Sometimes it does not find the device: unplugging and plugging in often helps.

If we just do the naive thing and cargo add iroh, we get a lot of compile errors. It turns out that while the ESP32 platform espidf is an unix, it does not support some advanced features like cmsg. Several symbols in the ESP32 specific libc are just not there. Also, many 32 bit architectures used for embedded devices don't have 64 bit atomic support.

We will make sure to support ESP32 from iroh main in the future, but for now you will have to use a special branch of iroh.

Now let's do a minimal iroh endpoint setup and see what happens. An ESP32 is an incredibly constrained environment. Every thread requires its own stack, so we will manually set up a single threaded tokio runtime instead of using async fn main().

There is some setup needed before the runtime can even start, so using tokio::main is not an option even if you configure a single threaded runtime.

Inside the runtime, we will just create an endpoint.

Missing symbols

When we compile this, we get a linker error. ESP32 does not provide a symbol that one of the iroh dependencies needs.

We don't care that much about the hostname, so we can just define a noop implementation:

Once we do that, compilation proceeds a bit further. We get to linking. But the troubles don't stop.

Binary size issues

Our binary is too large even in release mode to fit on the ESP32. But not by much. So we can just enable link time optimizations for release builds to get below the limit in Cargo.toml. While we are at it, we also optimize for size.

Unfortunately this will lead to even longer build times than normal release builds. But there is nothing we can do about it, and in any case flashing is even slower.

After enabling lto, we finally get to flash the program on the ESP32, which takes a while. We are almost at the size limit (88.53%).

Once flashing is complete we are greeted with a runtime error:

What is it this time? Asking your favourite coding LLM reveals that we need to register an eventfd VFS, which is used by the tokio runtime.

Memory

After flashing this change, we get a little bit further. This time we have a problem with malloc failing. Guru Meditation Error? Somebody likes Amiga it seems...

The default configuration uses only the internal memory of the ESP32. But that is very little. You can see a list of memory ranges during startup:

While it would be a fun challenge to try to get iroh to work with only internal memory, for now we will just use the external memory. While we're at it we will also increase the stack size.

sdkconfig.defaults:

We have configured all dynamic allocation to use the external SPIRAM, and this has allowed us to increase the stack size generously.

Enabling external memory makes our memory problems go away for now:

Crypto provider

After all these rather tedious problems, we finally get to something interesting. The tokio runtime starts, and even the iroh endpoint init runs.

Now we get the following error:

Published iroh by default uses ring as the crypto provider. But ring is a C dependency with lots of platform-specific assembly code, which does not work on ESP32.

There is an alternative crypto provider aws-lc-rs, but it has the same problem — it wraps AWS-LC, a C library with platform-specific assembly that does not support the Xtensa architecture.

So what do we do? Rustls provides pluggable crypto providers, and the latest version of iroh makes sure to always use the configured provider.

So now we would have two options. Implement a rust only crypto provider, or implement a crypto provider that uses the ESP32 built in hardware acceleration.

The latter would be the right thing to do for a production system, but for now we are going to just do a pure rust version.

Since we are already very close to the binary size limit, we will only implement the absolute minimum number of cryptographic primitives that we need for iroh to work, and even take some shortcuts.

There is a crate rustls-rustcrypto that provides glue between rustls and rustcrypto. Due to binary size issues we had to fork it and feature gate the various implemented algorithms. For iroh itself we only need TLS13_AES_128_GCM_SHA256 and X25519.

Now all that is needed is to add some glue code to make the crypto providers work with QUIC, and configure the global crypto provider.

WiFi

After all this ceremony, we get a bit further.

So we have the problem that TCP/IP does not work. But how is it supposed to work anyway? The ESP32 does not have a wired network card. What it does have is WiFi.

We need to set up WiFi, and also connect to an access point. This is an embedded project, so we are going to include the WiFi credentials into the binary. We don't want to hardcode them in the repo, so we configure them using an environment variable WIFI_CONFIG that is set at compile time.

While we are at it, we will also make sure the system time is set so certificates we use have somewhat correct times. ESP32 has built in SNTP support that we just need to enable.

Normal iroh stuff

At this point the endpoint setup works. Now all that remains to be done is to add a simple echo protocol and an accept loop.

Also we will use a compile time environment variable IROH_SECRET to allow configuring the endpoint id.

We also have the endpoint print a short and long ticket on the debug output, so we can try dialing it either using an ip address or address lookup.

To test this, we have a client in the repo for this blog post that uses published iroh from crates.io.

And that's it! We got a complete iroh endpoint running on a tiny device, that we can talk to from any iroh endpoint.

Next steps

For my home automation project, the next step is to finally wire up some sensors and actuators. I am using two DHT22 temperature and humidity sensors for the sensor part, a LED display, and a solid state relay to switch a 220V load.

This is where the ESP32 shines: you can wire up all kinds of sensors and actuators in minutes to the numerous GPIO ports. For example you can drive most servos directly from the ESP32, which has built in PWM support.

And ESP32 boards are cheap and small enough that you can fit a complete project in a tiny box.

For the iroh project, this experiment revealed a number of places where we can reduce dependencies, several of which already made it to iroh main. We will make sure that iroh main compiles on 32 bit embedded architectures, and that expensive dependencies are optional.

In this project we succeeded in running iroh on an ESP32 with just 4 MiB of flash and 4 MiB of SPIRAM. But more powerful variants are also available and cheap, if you need some heavy non-iroh dependencies or memory for e.g. image processing.

Trying it out

For a hobby project I would suggest getting one of the more powerful variants. Playing with sensors is fun, waiting for lto compilation for every build is not.

The project for this blog post is https://github.com/n0-computer/iroh-esp32-example. It uses a patched version of iroh and its dependencies that works for this example. We will gradually add stable embedded systems support to iroh.

If you have a special requirement for a commercial project, talk to us.

Iroh is a dial-any-device networking library that just works. Compose from an ecosystem of ready-made protocols to get the features you need, or go fully custom on a clean abstraction over dumb pipes. Iroh is open source, and already running in production on hundreds of thousands of devices.
To get started, take a look at our docs, dive directly into the code, or chat with us in our discord channel.