How to bring up the Linux Kernel on a new platform | WerWolv

Introduction

Many embedded systems these days run Linux as their operating system. Generally because it’s a great foundation to run anything you like on top of and because many great developers and manufacturers already took care of writing drivers for all kinds of hardware components.

Even though this makes building the finished project much easier, if you’re dealing with custom hardware, you will most likely still have to bring up the Linux kernel on your own initially. This post is about just that: Setting up the bare minimal to get Linux running on a new platform. In this case, the new platform is not a new PCB but a minimal, emulated RISC-V CPU.

Getting Linux to run

Even though Linux is a very complex piece of software, it doesn’t actually have that many requirements to run. The only thing it really needs in terms of hardware is:

• A CPU with an MMU
• Enough RAM to, at least, load the Kernel, a Device Tree and an initramfs.
• A periodically lapsing system clock timer

Writing an Emulator

These things are what I ended up implementing in about 2000 lines of C++ code. For the CPU I implemented the RV32IMA instruction set which has all the things Linux needs to run. For the Hardware, I implemented a simple RAM peripheral, the SBI interface for timers and a UART peripheral based on the WD8250 chip which is well supported and is really simple to implement.

Documenting the entire process of writing the emulator is a bit out of scope for this post but the full implementation can be found here: WerWolv/riscv-emulator.

Using this emulator, the final hardware definition looks like this:

struct Hardware {
  Hardware() {
    // Map peripherals into the address space
    cpu.address_space().map(0x0000'0000, &ram);
    cpu.address_space().map(0xF400'0000, &uart8250);
    // Hook up a RISC-V compatible MMU as the address translator
    cpu.address_space().add_address_translator(&riscv_mmu);
    // Configure the UART peripheral to print output to the terminal
    uart8250.output_callback([](std::uint8_t c) {
      putchar(c);
      fflush(stdout);
    });
    // Load the Linux Kernel to the start of the RAM
    ram.write(0x00, LinuxKernel);
    // Load the Device Tree to the end of the RAM
    constexpr static auto DeviceTreeBlobLoadAddress = 512_MiB - 1_MiB;
    ram.write(DeviceTreeBlobLoadAddress, DeviceTreeBlob);
    // Put the device tree address into register a1 to emulate the way
    // the bootloader would do it on a real system
    emulator.cores()[0].a1() = DeviceTreeBlobLoadAddress;
    // Load the initramfs in front of the device tree
    // but at a fixed address we can use later
    constexpr static auto InitRamFsLoadAddress = 0x1F700000;
    ram.write(InitRamFsLoadAddress, InitRamFs);
    // Start up cpu
    cpu.power_up();
  }
  auto step() -> void {
    cpu.step();
  }
private:
  riscv::Cpu<1> cpu;
  dev::riscv::MMU<std::uint32_t> riscv_mmu;
  dev::Ram ram(512_MiB);
  dev::UART8250 uart8250;
};

This, of course, currently does absolutely nothing since there’s no code for it to execute. To get Linux running, we need to first configure and compile the Linux Kernel for our platform and then populate those LinuxKernel, DeviceTreeBlob and InitRamFs arrays with the correct data.

To compile anything to run on the target, the first thing we need is a toolchain. Compiling it by hand is generally a bit of a pain since you need to get the correct version of various different tools and libraries and then configure and compile them in the correct order (sometimes even multiple times with different configurations!). Luckily for us though, the crosstool-ng Project exists and handles all of that for us. With it we can simply configure the toolchain we need and then let it do all the work.

Since we’re planing on building an application that later on runs on 32-bit RISC-V Linux, we need a toolchain that targets riscv32-unknown-linux-gnu. This will give us a GCC compiler that can compile C code for our target platform as well as glibc as the C standard library implementation to interface with the Kernel.

.
$ ./bootstrap
INFO  :: *** Generating package version descriptions
...
INFO  :: *** Done!
$ make
gmake[1]: Entering directory 'crosstool-ng'
...
GEN      ct-ng
$ ./ct-ng riscv32-unknown-elf
...
Now configured for "riscv32-unknown-elf"
$ ./ct-ng menuconfig

This will open a visual configuration menu where we can change various options for the toolchain.

In here, the main things that needs to be changed here are:

• Enable Target options -> Use the MMU
• Set Operating System -> Target OS to Linux
• Set C library -> C library to glibc
• Enable C compiler -> C++

Save the configuration and then run ./ct-ng build to start the compilation process. This can very well take up to an hour or more depending on your machine, on mine it took about 11 minutes. After it’s done, you should have a fully working riscv32-unknown-linux-gnu toolchain installed in ~/x-tools/riscv32-unknown-linux-gnu that you can use to compile the Linux Kernel and any other software for your target platform.

Compiling the Linux Kernel

Now that we have a toolchain, we can move on to compiling the Linux Kernel. After cloning the torvalds/linux repository, the first thing we need to do is to create a configuration for our platform. Since our platform is quite minimal and doesn’t have any special hardware, we can just use the default RISC-V configuration as a base and then modify it to fit our needs. To do that, we can use ARCH=riscv make menuconfig to once again open a visual configuration menu.

The main thing we need to change here is to get the Kernel to be built for 32 bit RISC-V instead of 64 bit. For that the following options need to be changed:

• Platform type -> Allow configurations that result in non-portable kernels option needs to be enabled
  • Now the Base ISA option below can be set to RV32I
• All the extension support options need to be disabled as they’re not implemented in the emulator.
• Boot options -> UEFI runtime support needs to be disabled
  • This is necessary only so the Emit compressed instructions when building Linux option can be disabled as our emulator doesn’t support the compressed instruction set.
  • Under Target options, the Generate code for the specific ABI option needs to be set to ilp32 and Architecture Level to rv32ima to prevent the compiler from using compressed instructions when building glibc and our init program later on.
• Under Kernel Features, I had to enable the SBI v0.1 support for our Timer and Build a relocatable kernel since that allows us to load the Kernel at any address in memory which is quite nice for our use case.
• Finally we can disable Virtualization, Enable loadable module support, Networking support and Cryptographic API since we don’t need any of that and it just bloats the Kernel and increases boot time.
• If you like, many of the Device Drivers can also be disabled since we don’t have any of that hardware. I left everything standard there though since it’s quite a few options to go through

Everything else can be left as it is. After saving the configuration, we can compile the Kernel using the following command:

$ make -j    \
  ARCH=riscv \
  CROSS_COMPILE=/path/to/riscv32-unknown-linux-gnu-

If the make process asks questions if you want to enable or disable any other options, you can simply answer n to all of them.

After the compilation is done, you should have a arch/riscv/boot/Image file that contains the compiled Kernel. This is the final file that can be executed.

Compiling a init program

Using the same compiler we can also compile a simple init program that will be executed by the Kernel as soon as it has booted and reached userspace. In a real world embedded system, this would generally be something like busybox's init binary but it can really be any ELF executable. I simply wrote a small C program that prints some text to the console.

#include <stdlib.h>
#include <stdint.h>
#include <stdio.h>
#include <fcntl.h>
#include <unistd.h>
int main() {
  // Open /dev/kmsg to write logs directly to the Kernel Log
  int kmsg = open("/dev/kmsg", O_WRONLY);
  if (kmsg == -1) {
    return EXIT_FAILURE;
  }
  // Print some message
  dprintf(kmsg, "Hello World from \033[32mLinux Userspace\033[0m!\n");
  // Loop forever to prevent the init process from exiting
  while (true) {
    sleep(1);
  }
  // Close the file handle
  close(kmsg);
  return EXIT_SUCCESS;
}

To make creating the initramfs a bit easier, we can compile the init program as a statically linked binary. This way we don’t have to worry about any shared library dependencies and can just put the binary itself into the initramfs without needing to include any additional files.

$ riscv32-unknown-linux-gnu-gcc -static init.c -o init

Creating an initramfs

An initramfs or Initial RAM File System is usually just a cpio archive that contains all files and directories that should be available to the system after it boots. It is mounted by the Kernel to / and usually contains things like /bin, /dev and /proc but it can really contain anything you want.

You can create a simple initramfs yourself by just creating a directory with the structure you want and then using the find and cpio commands to create the archive. However, building a reproducible initramfs this way can be difficult due to permissions, timestamps and other metadata that cpio includes in the archive. Instead, we can use a tool that was built automatically when we compiled the Kernel: gen_init_cpio. This tool takes in a textual description of the initramfs structure and content and then generates a cpio archive from it. The description file looks something like this:

.
# /bin
dir /bin 755 0 0
file /bin/init init 755 0 0
# /dev
dir /dev 755 0 0
nod /dev/kmsg 644 0 0 c 1 11
nod /dev/initrd 644 0 0 b 1 250
nod /dev/console 600 0 0 c 5 1
nod /dev/null 666 0 0 c 1 3
nod /dev/ram 644 0 0 b 1 0
nod /dev/root 644 0 0 b 4 0
nod /dev/ttyS0 660 0 0 c 4 64
# /proc
dir /proc 755 0 0

This file can be then finally compiled into a cpio archive using the gen_init_cpio tool like this:

$ gen_init_cpio initramfs.txt -i initramfs.cpio

Writing a Device Tree

A device tree is a data structure that describes the hardware of the system to the Linux Kernel. It is used by the Kernel to know which drivers to load and how to configure them.

Let’s start with the bare minimum and build up from there.

/dts-v1/;
/ {
  #address-cells = <0x01>;
  #size-cells = <0x01>;
  model = "WerWolv's Emulator";
  cpus {
    #address-cells = <0x01>;
    #size-cells = <0x00>;
    timebase-frequency = <65000000>;
    cpu@0 {
      device_type = "cpu";
      compatible = "riscv";
      reg = <0x00>;
      riscv,isa = "rv32ima";
      mmu-type = "riscv,sv32";
      interrupt_controller: interrupt-controller {
        #interrupt-cells = <0x01>;
        #address-cells = <0x00>;
        compatible = "riscv,cpu-intc";
        interrupt-controller;
      };
    };
  };
  memory@0 {
    device_type = "memory";
    reg = <0x00 (512 * 1024 * 1024)>;
  };
}

This configuration is enough to describe the CPU and its timer to the Kernel. However, what’s still missing is all the peripherals. These don’t go under the cpus node but instead under the soc node

/ {
  // ...
  soc@F0000000 {
    #address-cells = <0x01>;
    #size-cells = <0x01>;
    compatible = "simple-bus";
    ranges = <0x00 0xF0000000 0x10000000>;
    serial@4000000 {
      compatible = "ns8250", "ns16550";
      reg = <0x4000000 0x100000>;
      interrupt-parent = <&interrupt_controller>;
      interrupts = <0x01>;
      no-loopback-test;
      clock-frequency = <5000000>;
    };
  };
};

The final thing that’s needed is configuration for the Kernel itself:

/ {
  // ...
  aliases {
    serial0 = "/soc@F0000000/serial@4000000";
  };
  chosen {
    bootargs = "earlycon console=/dev/ttyS0 rdinit=/bin/init root=/dev/initrd";
    stdout-path = "serial0";
    linux,initrd-start = <0x1F700000>;
    linux,initrd-end = <0x1FEFFFFF>;
  };
};

The important bit here are the values in the chosen node.

• The bootargs property defines the kernel command line. Here we tell the Kernel that we want to use the earlycon feature to get log output as quickly as possible, it will do that by sending all logs to the serial0 device which is the alias we defined for our UART. We also tell the Kernel to execute our /bin/init program from the initramfs as the init process as soon as it has booted and reached userspace.
• linux,initrd-start and linux,initrd-end define the location of the initramfs in memory. That’s where we copied the initramfs we built previously to in the emulator.

Now that we have the full Device Tree ready, we can compile it to a Device Tree Blob (.dtb file) using the dtc compiler. This is the final file that can be loaded by the Kernel.

$ dtc -I dts -O dtb -o riscv-emulator.dtb riscv-emulator.dts

Putting it all together

The 3 files we built now can be bundled into the emulator binary and then we can finally run the emulator and see Linux booting!

constexpr std::uint8_t LinuxKernel[] = {
    #embed "Image"
};
constexpr std::uint8_t DeviceTreeBlob[] = {
    #embed "device_tree.dtb"
};
constexpr std::uint8_t InitRamFs[] = {
    #embed "initramfs.cpio"
};

$ ./command
[    0.000000] Booting Linux on hartid 0
[    0.000000] Linux version 7.0.0-00166-g0f0013213293 (werwolv@fedora) ...
[    0.000000] Machine model: WerWolv's Emulator
...
[   54.748992] Run /bin/init as init process
[   54.814251] Hello World from Linux Userspace!

Running it in the Browser

Now that we have a self-contained executable that we can run to get the Linux Kernel to boot, there’s nothing stopping us from also getting it to run in a Browser. For that, we can cross-compile the binary to WebAssembly and call into it from JavaScript.

Click on the Startup Machine button below to boot up the emulator.

Wrapping up

We now have a working Linux system running on our fantasy machine! While a few things here might feel a bit contrived, the general process is exactly the same for a real piece of hardware. Though instead of just choosing where things is mapped into memory, you’d instead consult the datasheet of your SoC.

Also things like the Kernel or the initramfs of course can’t just exist in the memory magically but need to be put there instead by a Bootloader. On embedded systems this is usually something like u-boot or a proprietary thing provided by the chip manufacturer. There, you might need to further bundle up the Kernel, Device Tree and initramfs in some custom file format that the Bootloader can understand then. Check out the Android Bootloader Manual if you’d like to read about a real-world example of this.

As always, the code that was used to get all of this running can be found on my GitHub Page: