This post is the first part of a three part series on concurrency in Serene’s runtime.
- Choosing the building blocks (this post)
- Fibers, the scheduler and the reactor
- A tiny HTTP server
As you may know, I’m working on a programming language called Serene. It has a runtime library that provides the crucial bits every program needs while it’s running. The runtime library is perfectly usable outside Serene too; in fact, I use it in several of my own tools that have nothing to do with the language. I’ve written before about how it manages memory with a tiny arena allocator. This time I want to talk about a much bigger part of the runtime: fibers, the scheduler, the Reactor, and how they let a Serene program do many things at once without tying themselves in knots.
Note: At the time of writing, Serene’s runtime only supports
x86_64-linux. Porting it to other platforms and architectures is straightforward, and it’s on my to-do list. Contributions are always welcome.
The problem
Concurrency is one of the hardest1 2 problems in systems programming. Once several independent pieces of work can run at the same time, the number of possible execution orders explodes, and so does the number of bugs hiding inside them. Race conditions, deadlocks, starvation, and subtle ordering bugs are all consequences of the same underlying reality: your program is no longer executing one instruction after another in a predictable sequence.
Fortunately, modern programming languages have spent decades developing abstractions that make concurrent programming far more approachable. Today, most programmers don’t need to think directly about schedulers, event loops, or synchronization primitives every time they write a network service. Those abstractions exist because countless engineers have already wrestled with the underlying complexity and turned it into reusable building blocks.
As someone building a programming language, I don’t get to take those building blocks for granted—I have to choose them. That means understanding the trade-offs each abstraction makes, deciding which ones fit Serene’s goals, and sometimes discovering that none of the existing choices are quite the right fit.
Abstractions and building blocks
The good news is that I don’t have to invent concurrency from scratch. Over the last few decades, people much smarter than me have explored a wide range of concurrency models, each making different trade-offs between simplicity, performance, and flexibility. My job isn’t to create a new abstraction, it’s to choose the one that best fits Serene.
To make those trade-offs concrete, let’s use a web server as an example. Every incoming client represents another unit of work that has to wait for network IO while still allowing the server to make progress on other requests. Different concurrency models solve that problem in very different ways, and the differences become much clearer when they’re all tackling the same workload.
We’ll start with the oldest and most obvious approach.
Threads
A common first step for concurrency is the OS thread: one thread per client, each thread blocks on its own socket, and the kernel does the multiplexing. That model is straightforward and - at small scale - hard to beat. You get real parallel execution on multiple cores without having to build much infrastructure.
But the “one thread per client” approach stops scaling once you move from a handful of clients to many thousands. The pain is cumulative:
Memory footprint (stacks). Every thread needs its own stack. Linux’s default stack size is often around 8 MB, which is fine if the thread is actually working - but for a web server, most connections are idle most of the time. Multiply that default by tens of thousands of threads and you end up reserving enormous amounts of memory for stacks that aren’t doing anything most of the time. The stack size becomes your practical limit on how many clients you can keep alive.
Locking and synchronization (preemption). Threads are preemptive: the kernel can interrupt a thread at any point. That’s good for fairness, but it’s bad for shared-state code. If two threads might touch the same data, they can be interleaved at arbitrary instruction boundaries, so shared access must be guarded. Locks are the price of that safety, and they’re also where complexity, contention, and “why does it deadlock only in production?” bugs tend to appear.
Context switch overhead. Switching between threads requires kernel involvement and comes with real costs: saving/restoring registers and disrupting cache and TLB locality. A single context switch isn’t a big deal; thousands of them per second add up into CPU time spent on orchestration instead of useful work.
Loss of control over scheduling. The kernel owns the scheduling policy. In a runtime, you generally want tighter integration between your scheduling decisions and your IO model. With OS threads, you can’t cheaply steer execution toward a particular peer, and you can’t teach the scheduler about your runtime’s semantics or how your blocking/unblocking works. And since threads are preemtive, trying to schedule os threads on user space, is a fruitless idea.
Also, there’s also a per connection “startup tax” too. Creating a thread is a syscall that allocates stack space and initializes kernel bookkeeping. In a one thread per request design, that cost repeats for every new connection. A thread pool reduces the worst of this, but it’s also a concession: you’re now managing a limited pool of OS threads because creating more is too expensive. It works fine for many use cases but still, not the optimal choice. For example, one need to be careful with the type IO operations in a thread pool, blocking IO operations can starve a thread pool.
Threads don’t disappear entirely. The runtime still needs them, we’re just going to use them differently.
Callbacks and event loops
If threads don’t scale, the next idea is to stop leaning on so many of them. Instead of one thread blocking per client, you keep a single thread spinning in an event loop, ask the kernel “which of these thousands of sockets is ready right now?”, and run a small callback for each one that is. This is essentially how libuv that powers NodeJS, works. One thread can shepherd tens of thousands of connections, the memory footprint is tiny, and the scaling numbers are genuinely impressive. The catch this time isn’t performance, it’s what the model does to your code:
Inverted control flow (callback hell). A request handler that would read cleanly from top to bottom (read the request, process it, reply) gets chopped into a separate callback for each step. Everywhere you’d naturally wait, you instead register “call me back when this is ready” and return. The shape of the computation is turned inside out, and the deeper it nests, the worse it reads. People call it “callback hell” for a reason.
You lose the stack (manual state). The stack is what normally remembers where you were and what you were doing, halfway down a call chain. Once you return to the loop, that memory is gone, so you rebuild it by hand. The locals that would have lived on the stack now live in heap allocated state objects that you thread from one callback to the next. You end up hand rolling the bookkeeping the CPU used to do for free.
Errors and lifetimes get slippery. Without a stack, there’s no natural place for an error to unwind to, so error handling gets threaded through the callbacks by hand, and it’s easy to drop one. Lifetimes go the same way. Working out who owns a buffer and when it’s safe to free it, across a chain of callbacks that fire at unpredictable times, is precisely the kind of thing that leaks or frees too early. It might not be an issue for a language with a GC, in the absence of one it becomes a problem.
Still one core (no free parallelism). An event loop is single threaded by nature. To actually use the machine, you run several loops, one per core, and now you’re coordinating between them again, which drags a slice of the thread concerns right back in — that being said, event loops are single threaded in the sense that the execution of callbacks happens on the same thread, but usually they have this offload thread pool for tasks that have to block.
The event loop got the scaling right and the ergonomics wrong. It proved you don’t need a thread per client, you just need somewhere to park “what to do when this is read” and a loop to run it. Hold on to that idea, the IO Reactor in the next post is exactly that loop done properly. The open question it leaves behind is whether we can keep the scaling and still get our straight line code back.
async/await
Modern languages clean this up with async/await. You write code that looks straight line, and the compiler quietly
rewrites every function that can suspend into a state machine. It’s clever and it’s efficient and honestly it’s pleasant
to use most of the time.
But it comes with a property people call function colouring, and it bites. Just read the article “What Color is Your
Function?” by Bob Nystrom who explains it
pretty well. A function that can suspend is a different colour from one that can’t, and only a same coloured function
can call it. That colour spreads outward through your whole call graph, and it stops dead at any boundary the compiler
doesn’t control. Think about what that means for a language like Serene. I have a JIT that emits machine code the
compiler never saw. I have an FFI that calls into foreign C libraries the compiler will never rewrite. The moment one of
those frames is on the stack and wants to wait, async/await has nothing to offer, because it only works on code it
was allowed to rewrite. That’s a dealbreaker for me.
This isn’t a criticism of async/await. It’s the wrong tool for my constraints.
What I’m looking for
After looking at all three approaches, the runtime needs something with four properties:
- Straight-line code.
- Cheap enough to create by the thousands.
- A real call stack.
- The ability to suspend regardless of whether the stack contains JIT code or foreign C.
Only one abstraction checks all four boxes.
Why fibers, and why they fit Serene
Three building blocks, and each one taxes something I care about. Threads are too heavy and drag locks in with them.
Callbacks shred the story my code was telling. async/await colours every function and gives up at the first foreign
frame. So let me turn the problem around, describe the weapon I actually want, and then name it.
I want code that waits to be written as if it didn’t wait, one line after another, with a real stack that remembers the story. I want thousands of these things to be cheap. And I want it to work for any code on the stack, including JIT output and foreign C, not just code my compiler was allowed to touch. That set of wishes points at exactly one thing. Fibers.
A fiber is a unit of execution that the Serene runtime schedules, not the kernel. It’s a cooperative, lightweight thread that lives inside an operating system thread, and many of them share one such thread, taking turns. Three properties define it, and each one falls straight out of a constraint I actually have.
It’s cooperative. Nothing interrupts a fiber against its will. It runs until it chooses to yield or until it asks to wait. That means two fibers on the same worker never overlap, so a whole category of data races just evaporates, no lock required. This lines up with one of Serene’s guiding values, explicit over implicit. A fiber gives up control at points it named, not at some random instruction the scheduler picked.
It’s stackful. Each fiber carries a real machine stack, the actual chain of call frames. That’s the expensive choice compared to the state machine trick, but it’s the one that buys me everything. Because suspending a stackful fiber is a property of the stack, not of the functions that built it, I can suspend a fiber that’s three calls deep into a parser, ten calls deep into the evaluator, sitting on top of a foreign C frame, sitting on top of JIT emitted code, and it all freezes and thaws the same way. No colouring. No rewriting. It works for code the compiler never compiled, which is precisely what a language with a JIT and an FFI needs.
Its stack has a fixed size. This one is a consequence, not a preference. The operating system can quietly grow a thread’s stack, and goroutines can grow and even relocate theirs, but relocating a stack means going through it and fixing every pointer that pointed into it, and that needs a garbage collector and precise stack maps to know which words are pointers. Serene has neither. So a fiber’s stack is a fixed span it can never outgrow, and we’ll see in the next post how it stays safe.
None of this is new in the abstract. Green threads, coroutines, goroutines, and fibers are all variations on the same fundamental idea. What’s interesting isn’t that Serene has fibers—it’s why they look the way they do. Every property we’ve discussed is a consequence of the constraints the runtime has to live with: no garbage collector, no stack maps, and a runtime that must happily suspend and resume code it never compiled itself. Change those constraints, and I’d almost certainly make different design choices.
So that’s the abstraction chosen. In the next post we’ll stop talking about why fibers are the right fit and start looking at how they’re built. We’ll see how a fiber becomes a suspended call chain with its own stack, how the scheduler moves execution between thousands of them, and how the IO Reactor turns kernel events back into runnable fibers-all while keeping those design goals intact.
Until next time. :)