PSA: fork() considered harmful in macOS

Drew Gregory · Formal ·

13 min read Original article ↗

At Formal, we love proxies, and we’ve been spending a lot of time writing proxies that don’t just run in containers but also on endpoints. As a result, our Go code has been running more and more on macOS lately. It also has been growing more and more Swift CGo bindings!

Upon upgrading to macOS Tahoe 26.5.2, however, some customers were experiencing some bizarre high CPU usage from our endpoint. We grabbed the following profile via sample:

Call graph:
    2395 Thread_1014362: Main Thread   DispatchQueue_<multiple>
      2395 ???  (in formal-agent) 
        2395 ???  (in formal-agent)
          2395 fork  (in libsystem_c.dylib) + 112
            2395 _pthread_atfork_child_handlers  (in libsystem_pthread.dylib) + 76
              2395 nw_settings_child_has_forked()  (in Network) + 296
                2395 nw_path_release_globals  (in Network) + 164 
                  2395 NEFlowDirectorDestroy  (in libnetworkextension.dylib) + 64
                    2395 os_log_type_enabled  (in libsystem_trace.dylib) + 772
                      2395 _os_log_preferences_refresh  (in libsystem_trace.dylib) + 56

Total number in stack (recursive counted multiple, when >=5):

Sort by top of stack, same collapsed (when >= 5):
        _os_log_preferences_refresh  (in libsystem_trace.dylib)        2395

We ship a macOS Network Extension on the darwin build of our Go application, which explains the presence of nw and NE functions in our trace. Yet so many mysteries abound:

  • Why are all samples caught on this one trace?
  • What the heck are these _pthread_atfork_child_handlers doing?
  • Why is this taking up 98% CPU?

It seems like we were not the only ones to experience this problem: skaffold and GitHub Actions runners seemed to hit the same behavior.

Like any good mystery, we’ll try to follow along by running reproduction code. All of this code reproduced on an Apple M5 Max (thanks Formal! we do have quite the docker compose stack!) on macOS Tahoe 26.6.2.

Why _os_log_preferences_refresh?

It is a bit peculiar why one instruction in _os_log_preferences_refresh would be hogging all 2,395 CPU samples. Let’s identify the exact instruction at _os_log_preferences_refresh+ 56. Since this is in the dynamic library, even /usr/bin/true was sufficient to find the relevant instructions.

lldb --batch \
  -o 'target create /usr/bin/true' \
  -o 'image lookup -vn _os_log_preferences_refresh' \
  -o 'disassemble --name _os_log_preferences_refresh --count 24' \
  -o 'quit'
(lldb) target create /usr/bin/true
Current executable set to '/usr/bin/true' (arm64e).
(lldb) disassemble --name _os_log_preferences_refresh --count 24
libsystem_trace.dylib`_os_log_preferences_refresh:
libsystem_trace.dylib<+0>:  pacibsp
libsystem_trace.dylib<+4>:  sub    sp, sp, #0x50
libsystem_trace.dylib<+8>:  stp    x24, x23, [sp, #0x10]
libsystem_trace.dylib <+12>: stp    x22, x21, [sp, #0x20]
libsystem_trace.dylib <+16>: stp    x20, x19, [sp, #0x30]
libsystem_trace.dylib <+20>: stp    x29, x30, [sp, #0x40]
libsystem_trace.dylib <+24>: add    x29, sp, #0x40
libsystem_trace.dylib <+28>: mov    x19, x0
libsystem_trace.dylib <+32>: bl     0x18023d654    ; symbol stub for: __error
libsystem_trace.dylib <+36>: ldr    w22, [x0]
libsystem_trace.dylib <+40>: cbz    x19, 0x1802252d4 ; <+412>
libsystem_trace.dylib <+44>: ldr    x8, [x19, #0x18]
libsystem_trace.dylib <+48>: cbz    x8, 0x1802252d4 ; <+412>
libsystem_trace.dylib <+52>: add    x20, x8, #0x4
libsystem_trace.dylib <+56>: ldrb   w23, [x8, #0x2]
libsystem_trace.dylib <+60>: str    xzr, [sp, #0x8]
libsystem_trace.dylib <+64>: adrp   x8, 445723
libsystem_trace.dylib <+68>: ldr    x0, [x8, #0xa78]
libsystem_trace.dylib <+72>: cbz    x0, 0x180225198 ; <+96>
libsystem_trace.dylib <+76>: mov    x1, x20
libsystem_trace.dylib <+80>: bl     0x18023e594    ; symbol stub for: xpc_dictionary_get_dictionary
libsystem_trace.dylib <+84>: cmp    x0, #0x0
libsystem_trace.dylib <+88>: cset   w24, ne
libsystem_trace.dylib <+92>: b      0x18022519c    ; <+100>

So the guilty instruction is

libsystem_trace.dylib <+56>: ldrb   w23, [x8, #0x2]

The ldrb instruction shouldn’t be hogging CPU! How do we reproduce this?

Unfortunately, the Network Extension fork creation handler registration code is a bit hard to reproduce, but this _os_log_preferences_refresh path seems pretty pervasive in their Network Extension code. In fact, there is a similar bug report that hit the same leaf instruction of _os_log_preferences_refresh. Thankfully, it seems like this dylib isn’t changing very much across macOS versions! Although the reproduction case does not rely on the same fork creation handlers in our original sample trace, let’s check if the segfault is on the same instruction. The segfault was a bit flaky, so we added a loop here:

cat > /tmp/oslogfork.c << 'EOF'
#include <netdb.h>
#include <os/log.h>
#include <stdio.h>
#include <string.h>
#include <sys/wait.h>
#include <unistd.h>
static void lookup(void) {
    // This bug is pretty weird. Performing `getaddrinfo` with AF_UNSPEC
    // for a hostname whose DNS answer contains only A records fails in forked child processes
    // when the parent performed DNS resolution. Kudos to adamoffat for describing
    // the problem so well!
    struct addrinfo hints, *res = NULL;
    memset(&hints, 0, sizeof hints);
    hints.ai_family = AF_UNSPEC;
    hints.ai_socktype = SOCK_STREAM;
    // Let's hope api.stripe.com doesn't add an AAAA record for this reproduction case! IPv4 obsolescence is pretty far away :-)
    getaddrinfo("api.stripe.com", "443", &hints, &res);
    if (res) freeaddrinfo(res);
}
int main(void) {
    os_log_t log = os_log_create("com.example.repro", "repro");
    os_log(log, "parent");
    lookup();
    pid_t pid = fork();
    if (pid == 0) {
        lookup();
        _exit(42);
    }
    int st;
    waitpid(pid, &st, 0);
    if (WIFSIGNALED(st)) {
        fprintf(stderr, "signal %d\n", WTERMSIG(st));
        return 0;
    }
    fprintf(stderr, "flake: exited %d\n", WEXITSTATUS(st));
    return 1;
}
EOF
clang -o /tmp/oslogfork /tmp/oslogfork.c
for i in $(seq 1 50); do
  echo "=== $i ==="
  /tmp/oslogfork && break
done
sleep 2

This bug is a bit unreliable, so let’s run it up to 50 times until it crashes. Then, let’s analyze the crash report for the trace!

REPORT=$(ls -t "$HOME"/Library/Logs/DiagnosticReports/oslogfork*.ips 2>/dev/null | head -1)
echo "REPORT=$REPORT"
lldb --batch \
  -o 'command script import lldb.macosx.crashlog' \
  -o "crashlog $REPORT"
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1)
  * frame #0: libsystem_trace.dylib`_os_log_preferences_refresh + 56 [synthetic]
    frame #1: libsystem_trace.dylib`os_log_type_enabled + 771 [synthetic]
    frame #2: Network` [synthetic]
    frame #3: Network` [synthetic]
    frame #4: Network` [synthetic]
    frame #5: Network` [synthetic]
    frame #6: Network` [synthetic]
    frame #7: Network`nw_path_access_agent_cache + 187 [synthetic]
    frame #8: Network` [synthetic]
    frame #9: Network` [synthetic]
    frame #10: Network`_nw_path_update_is_viableTm + 83 [synthetic]
    frame #11: Network`nw_path_snapshot_path(NWConcrete_nw_path*) + 67 [synthetic]
    frame #12: Network`nw_path_evaluator_evaluate(NWConcrete_nw_path_evaluator*, int*) + 2159 [synthetic]
    frame #13: Network`nw_path_create_evaluator_for_endpoint + 71 [synthetic]
    frame #14: Network`nw_nat64_v4_address_requires_synthesis + 239 [synthetic]
    frame #15: libsystem_info.dylib`_gai_nat64_second_pass + 275 [synthetic]
    frame #16: libsystem_info.dylib`si_addrinfo + 1387 [synthetic]
    frame #17: libsystem_info.dylib`getaddrinfo + 171 [synthetic]
    frame #18: oslogfork`lookup + 83 [synthetic]
    frame #19: oslogfork`main + 167 [synthetic]
    frame #20: dyld`start + 6991 [synthetic]

This is consistent with the EXC_BAD_ACCESS error found in the skaffold GitHub issue. Yet why would an EXC_BAD_ACCESS cause both the child process to be in a busy loop while the parent process is hanging? And why don’t we see any evidence of the SIGSEGV in our sample?

It is now worth identifying where our application code is forking. Our application code doesn’t really perform fork()s: we’re using Go, and it’s 2026! The culprit is exec.Command().Start(), which under the hood uses a fork/exec pattern!

// StartProcess wraps [ForkExec] for package os.
func StartProcess(argv0 string, argv []string, attr *ProcAttr) (pid int, handle uintptr, err error) {
	pid, err = forkExec(argv0, argv, attr)
	return pid, 0, err
}

In addition, Go adds a sigblock here:

func syscall_runtime_BeforeFork() {
	gp := getg().m.curg

	// Block signals during a fork, so that the child does not run
	// a signal handler before exec if a signal is sent to the process
	// group. See issue #18600.
	gp.m.locks++
	sigsave(&gp.m.sigmask)
	sigblock(false)

	// This function is called before fork in syscall package.
	// Code between fork and exec must not allocate memory nor even try to grow stack.
	// Here we spoil g.stackguard0 to reliably detect any attempts to grow stack.
	// runtime_AfterFork will undo this in parent process, but not in child.
	gp.stackguard0 = stackFork
}

This means that code that runs during the fork() from the go runtime’s perspective is blocked from handling any signals. Unfortunately, ARM architectures seem to have a convention of resetting the program counter to the erroring instruction, so we get stuck in an infinite loop of kernel errors and have processes that hang at 98% CPU. Since finding the exact conditions to register the nw_settings_child_has_forked handler in a way that causes the segfault was tricky, we tied together:

  • The getaddrinfo() segfault reproduction case.
  • A custom pthread_atfork registration of the reproduction case a handler.
  • The pthread_sigmask that reproduces the Go runtime’s sigblock function that runs before fork() and resets the sigmask after fork.
clang -arch arm64e -o /tmp/oslogatfork -x c - <<'EOF'
#include <netdb.h>
#include <os/log.h>
#include <pthread.h>
#include <signal.h>
#include <stdio.h>
#include <sys/socket.h>
#include <sys/wait.h>
#include <unistd.h>
static void lookup(void) {
    // Use the same repro as before. This time, however, the SIGSEGV won't be caught!
    struct addrinfo hints = {
        .ai_family = AF_UNSPEC,
        .ai_socktype = SOCK_STREAM,
    };
    struct addrinfo *res = 0;
    getaddrinfo("api.stripe.com", "443", &hints, &res);
}
static void child_atfork(void) {
    lookup();
}
int main(void) {
    setbuf(stderr, 0);
    os_log_t log = os_log_create("com.example.repro", "repro");
    os_log(log, "init");
    lookup();
    // To replicate the fork behavior more closely, register the lookup code as a fork handler.
    pthread_atfork(0, 0, child_atfork);
    // The pthread_sigmask is what `sigblock` is doing under the hood in the referenced Go code.
    sigset_t all, old;
    sigfillset(&all);
    pthread_sigmask(SIG_SETMASK, &all, &old);
    pid_t p = fork();
    if (p < 0)
        return 1;
    if (p == 0) {
        // The "child" process in our exec.Command.Start() example. In practice, an `exec()` would
        // happen here, but we can exit instead.
        _exit(42);
    }
    // Although this doesnt matter for our repro, after forking the parent reverts
    // the sigmask that blocks signal handling in the child. This is what Go does.
    pthread_sigmask(SIG_SETMASK, &old, 0);
    fprintf(stderr, "child pid=%d\n", p);
    int st = 0;
    // The key line that our poor formal-agent gets stuck on: waiting
    // for a child process that will never complete!
    waitpid(p, &st, 0);
    if (WIFSIGNALED(st))
        fprintf(stderr, "signal %d\n", WTERMSIG(st));
    else
        fprintf(stderr, "flake: exited %d\n", WEXITSTATUS(st));
    return 0;
}
EOF
for i in $(seq 1 100); do
  echo "=== $i ==="
  /tmp/oslogatfork
done

After a couple invocations in that for loop, we got a hung process with a child PID.

=== 1 ===
...
=== 7 ===
child pid=71655

In a separate terminal window, we ran:

$ sample 71655
Call graph:
    8499 Thread_916434   DispatchQueue_1: com.apple.main-thread  (serial)
      8499 start  (in dyld) + 6992
        8499 main  (in oslogatfork) + 228
          8499 fork  (in libsystem_c.dylib) + 112
            8499 _pthread_atfork_child_handlers  (in libsystem_pthread.dylib) + 76
              8499 child_atfork  (in oslogatfork) + 16
                8499 lookup  (in oslogatfork) + 76
                  8499 getaddrinfo  (in libsystem_info.dylib) + 172
                    8499 si_addrinfo  (in libsystem_info.dylib) + 1388
                      8499 _gai_nat64_second_pass  (in libsystem_info.dylib) + 276
                        8499 nw_nat64_v4_address_requires_synthesis  (in Network) + 240
                          8499 nw_path_create_evaluator_for_endpoint  (in Network) + 72
                            8499 nw_path_evaluator_evaluate(NWConcrete_nw_path_evaluator*, int*)  (in Network) + 2160
                              8499 nw_path_snapshot_path(NWConcrete_nw_path*)  (in Network) + 68
                                8499 _nw_path_update_is_viableTm  (in Network) + 84
                                  8499 nw_path_access_agent_cache  (in Network) + 188
                                    8499 os_log_type_enabled  (in libsystem_trace.dylib) + 772
									  8499 _os_log_preferences_refresh  (in libsystem_trace.dylib) + 56

Total number in stack (recursive counted multiple, when >=5):

Sort by top of stack, same collapsed (when >= 5):
        _os_log_preferences_refresh  (in libsystem_trace.dylib)        8499

And sure enough, we saw a 98% CPU hanging process!

$ ps -p 71655 -o pid,%cpu,state,command
  PID  %CPU STAT COMMAND
75537  99.9 R+   /tmp/oslogatfork

How the hang happens: formal-agent waits on a child stuck in _pthread_atfork_child_handlers

The Bug: using fork()?

Using the Network framework and calling fork() in the same process seemed to be enough of a risk to trigger this bug. Upon looking at this further, there are plenty of other (1) related (2) bugs that involve calling fork() on macOS.

Unfortunately, it seems like this behavior is not going to be fixed in macOS anytime soon. The similar Apple Developer Forums post has the following response from Quinn:

I don’t have any good news for you here )-:

As you’re aware, the runtime environment you get when you call fork but not exec* is extremely restricted, and calling getaddrinfo is not something we officially support.

More directly:

If you’re creating a library, and thus have no control over the type of program that your library is loaded in, this isn’t viable given your platform requirements. That’s because, on Apple platforms, doing a fork without an exec is only safe if you stick with the very lowest level frameworks. High-level frameworks lean heavily into Mach, and run into problems because the Mach port namespace [1] is not copied to the child process.

If you find us quoting two forum posts not particularly authoritative, then you haven’t heard of the Quinn “The Eskimo” @ DTS @ Apple. Quinn Apple Developer Forum posts are, in our humble opinion, as authoritative as WWDC talks and official Apple documentation. There are whole GitHub repos, Hacker News threads, and 26-year-old interviews dedicated to Quinn.

This is a bit unfortunate, since lots of Go code uses os/exec to perform analogues of exec.Command().Start().

Why is Go using fork() on darwin?

One would expect this to be a highly pervasive problem:

  • Go uses the fork/exec pattern on every Start() invocation. Further, they perform sigmasks to block signal handlers during Go’s fork implementation.
  • The manpage of fork on darwin has a section specifying:

There are limits to what you can do in the child process. To be totally safe you should restrict yourself to only executing async-signal safe operations until such time as one of the exec functions is called. All APIs, including global data symbols, in any framework or library should be assumed to be unsafe after a fork() unless explicitly documented to be safe or async-signal safe. If you need to use these frameworks in the child process, you must exec.

There is a hint from the Go archives:

As a result, we’re not expecting changes from macOS nor the Go communities anytime soon to make these errors go away.

Our solution: remove all usages of exec.Command() in our Go codebase!

A posix_spawn rewrite

Quinn’s recommendation:

Correct. That’s because it’s not safe to use fork from Swift [1]. However, most of this stuff can be done via posix_spawn, or Process, both of which you can use from Swift.

Thankfully, since all of our fork() invocations were tied to exec.Command(), posix_spawn largely worked as a perfect replacement. Kudos to @orospakr for https://github.com/orospakr/spawnexec! We would also love to see an official implementation in the Go Standard Library someday.

Setting up an AST scanner

Unfortunately, it can be non-obvious when third-party Go packages call fork(). Fortunately, capslock came to the rescue! Although code that crosses a CGo boundary may still perform fork() without us catching it, in practice catching all sinks of CAPABILITY_EXEC map that capslock ships out of the box with a couple additions on our part caught all invocations that used fork() in our desktop app. The capslock analyzer also respects the build target so we only had to worry about darwin cases.

In addition, all of the paths that we caught we could generally:

  • Replace wholesale with a posix_spawn variant
  • Reason through the control flow that this will not trigger

Patching packages has not been required at this point.

Takeaways

This whole experience revealed a fragile Go/macOS contract that we weren’t aware of before. Frankly, the fork() contract itself seems somewhat shaky, and it’s pretty amazing that more macOS software doesn’t hit this limitation more frequently. Hooray for posix_spawn!

Have more macOS stories to share? Found yourself down your own Quinn rabbit holes? Come say hi! We are fortunate to work with Cursor, Notion, and Decagon and are actively hiring!