Working In Groups - NFHN Reader

- 2025-08-30

Wann hast du Geburtstag?

A little over a year ago, Bevy introduced a new "Working Groups" process, as part of an effort to overcome some of the issues we were having with RFCs. Cart discussed this briefly in the 5th birthday post, but I'm going to go into more detail about why we made this change, the ways I think it has succeeded and failed, and what conclusions I think the maintainers of other projects might draw from our little experiment.

A Problem Of Scale

Many successful open source projects suffer from the same problem: Massive contributor base, minimal structure. Take Bevy as an example. I believe the last time Charlotte counted, we had somewhere around 60 regular contributors (people who merged at least ten PRs per year). That's actually pretty good for a project of Bevy's size, but it only represents a tiny fraction of the larger contributor base. Apparently Bevy merged 9,795 PRs last year, from somewhere upwards of 350 different contributors.

Yowza! That's a lot of engineers.

Imagine if Bevy were a company. With that number of developers, it would certainly have multiple isolated departments run by their own vice-presidents, an army of middle-managers, maybe a sales team, and probably several project managers. But Bevy isn't a company. It has a project lead, one full-time staff-engineer/engineering-PM, three maintainers, and a loose unstructured confederation and 1200 other people¹ who are, by and large, doing their own thing.

Lack of structure isn't inherently a bad thing. Most features and fixes are uncontroversial and can be decomposed into bite-sized tasks that a single engineer can complete more or less on their own. As long as bureaucracy and friction are kept comparatively low, it's possible to achieve really high throughput for this sort of work. Everyone just does whatever looks most interesting, more or less on their own. It takes some light community organizing and speedy code review, but the work gets done and the project keeps trundling along.

However, not everything fits neatly into the "single-developer" mold. Invariably, projects appear which are too big or too controversial for a single developer to tackle. All the really important aspects of software development — standards writing, API design, and broad system architecture — inherently need some degree of collaboration and consensus-building. Work like this calls for a different set of tools.

More often than not, this boils down to some sort of RFC process.

More often than not, this is a mistake.

Everybody Gets RFCs Wrong

Now, I love a good RFC. But I have a problem with how most people approach writing RFCs, and how most RFC processes are administered.

All RFC processes — Python's PEPs, BitTorrent's BEPs, Bevy, and Rust's RFCs — are to some degree modeled off the original IETF RFC series. Here's how that process was originally described. These are excerpts from RFC 3, drafted in the summer of 1969:

The content of [an RFC] may be any thought, suggestion, etc. [...]

Notes are encouraged to be timely rather than polished. Philosophical positions without examples or other specifics, specific suggestions or implementation techniques without introductory or background explication, and explicit questions without any attempted answers are all acceptable.

The minimum length for [an RFC] is one sentence.

These standards (or lack of them) are stated explicitly for two reasons. First, there is a tendency to view a written statement as ipso facto authoritative, and we hope to promote the exchange and discussion of considerably less than authoritative ideas. Second, there is a natural hesitancy to publish something unpolished, and we hope to ease this inhibition.

Unfortunately, some fifty years later, many modern RFC processes seem to have forgotten these guidelines. Getting the first draft of an RFC published is usually an exhausting, sometimes excruciating process. Incomplete thoughts or designs, documents that collate useful information without making recommendations, and unanswered questions are often not accepted.

Take Rust as an example. Rust has, on the whole, a healthy skepticism of RFCs submitted by project newcomers. It is not a very open process. The Rust project also apparently uses something called a "pre-RFC" to vet documents before they reach the RFC stage; something akin to an RFC, but without the procedure. These "pre-RFCs" are, as far as I can tell, coupled with prototype compiler implementations, and shuffled around for informal review until deemed "ready for general consumption". This protective attitude is justified by the observation that

[...] once an RFC has been published, it actually very rarely evolves significantly, because the author becomes entrenched in the proposal they've made. ²

I don't begrudge Rust for adopting these procedures; they are a small team of experts navigating the development of a language and compiler that are at once extremely popular and extremely complicated. Aggressive vetting is obviously required. Still, to me as an outsider, this sounds like a red flag. RFCs are "Requests For Comments" after all. They are intended to stimulate discussion, and to be updated based on those discussions until consensus is reached. If authors are regularly becoming "entrenched" in their existing views, that is, in my layman's opinion, a bad sign.

Bevy had entirely different issues with its RFC process. Writing an RFC took ages, getting it through review took ages, and then getting it actually accepted took ages. Partly this was because reading and reviewing an RFC is a heck of a lot harder than reviewing code, and partly it was because it wasn't clear whose job it was to make the "yes/no" call.

The result was burnout. We got some great RFCs, but usually the author was so exhausted after getting the design approved that they couldn't manage the implementation. When things ended up being implemented, it often was much harder than expected, or the design had to be fundamentally altered. A few years on, most of Bevy’s accepted RFCs are now either not completely implemented or outdated.

These problems are not universal. It's possible you've participated in an RFC process that didn't suffer from these issues (in which case, please reach out, I'd love to hear more about it). But I'm willing to bet most people with any experience with RFCs will empathize with the issues I've described. Many RFC processes have issues like these. And almost always it's because they have neglected an important aspect of what made RFCs make sense.

If you haven't guessed it yet, that thing is the "Working Group".

It Was Always Working Groups

RFC 3, which I quoted in the previous section, doesn't talk about RFCs. It uses the term "NWG Note" instead, because originally RFCs were notes circulated to six members of the IETF Network Working Group.

I think this is super important context. Originally, an RFC was literally a "request for comments from the other five guys in the working group". The members of the NWG all knew each other. They were working on similar stuff, and they started the RFC series to track and discuss their ideas. If you take process out of a collaborative group setting, the whole thing kind of falls apart.

All of the Bevy RFCs had basically a single author, as far as I can tell. They mostly weren't writing for their peers, and they always didn't have buy-in before they started writing. Without collaborators, the main source of comments on RFCs was the maintainers, and often reviews could be quite sparse because of the difficulty of the material. To the degree that any RFC was successful, it was due in large part because other members of the community were willing to spend the time to become deeply involved. And that didn't always happen.

The lack of collaboration is, I think, why authors tended to burn out, and why RFCs tended to languish without implementations.

Honestly, I find this solo approach to RFCs very odd. After all, one of the neat things about working groups is that they are self-organizing. They spring up naturally after you get enough engineers in a room together, like mushrooms after a rain. People with similar interests naturally seek each other out if given the opportunity, and once two or three like-minded contributors get to talking, it's only a short step to making plans and collaborating.

So I'm surprised that we didn't see many “homegrown working groups" submitting proposals through the RFC process. Outside the RFC system, we see them everywhere! Many of Bevy's complex or controversial features were pushed forward by a team of three or four dedicated contributors, who could all review each other's work and collaborate.

The New Process

After much discussion, we settled on a new process that would re-emphasize the importance of collaboration on complex features, but which would allow maintainers to steer (or veto) the direction of the work. You can read the full set of guidelines here, but I will also briefly summarize them:

To form a new working group, three people must put together a minimal proposal (a few sentences) and ask for maintainer ascent.
If accepted, the maintainers give them a dedicated space to communicate (a Discord thread), and the working group is tasked with developing a design document (essentially an RFC) and submitting it to the maintainers.
If the design doc is accepted by the maintainers, they may proceed to implementation without having to justify their design choices during code review (expediting the process).
Anyone can join or leave the group at any point, and there is no appointed leader. The group is encouraged to write down as much as possible, so it is easy for new members to join.
Groups are formally disbanded by the maintainers after the implementation is completed, there are fewer than three active participants, the work stalls, or the direction is vetoed.

We hoped this process would:

Force people to check in with maintainers before starting work.
Allow the maintainers to limit the number of initiatives underway at once.
Help distribute the load over more people.
Ensure that there are always two people capable of doing internal peer review.
Allow design documents to change and shift during the implementation phase.
Direct bright-eyed new contributors to experts who can put them to work.

There was a second change that, in retrospect, was no less important. Previously, all our design documentation lived in GitHub discussions or as Markdown files within the RFC repo. But shortly before introducing working groups, we also created a quasi-official HackMD Org. HackMD is Google Docs for Markdown, and though it wasn't explicitly specified, most working groups ended up relying on HackMD to manage their associated design documents.

What Went Well

We've now been using this process for well over a year and have hosted 27 working groups in that time.

5 were for releases or game jams.
3 were otherwise process or community-focused.
Of the remaining 19 feature-oriented working groups:
- 8 are effectively completed (curves, color, picking, no_std, text, bsn, audio, relations)
- 2 were closed as unsuccessful (render-graph, decoupled-rendering)
- 2 have temporarily stalled due to lack of contributors (ptr, transforms)
- 7 are ongoing (cli, hotpatching, wesl, tilemaps, nonsend-data, observers, rendering-refactor)

Which means that, of the 19 feature-oriented working groups, 15 either ended up completed successfully or are on track to be completed. When I ran a similar tally in April, this stood at 11 groups out of 14. If we only consider the definitively ended groups, it's 8 successes and effectively 2 failures (I expect the two stalled groups to get back into motion soon).

These numbers are pretty encouraging: more than half the working groups we started last year reached a definitive conclusion, and three-fourths were successful. By contrast, of the 74 Bevy RFCs, 32 have stalled with no expectation of completion and 33 were closed without being accepted. Only 9 were ever accepted, and 5 took two or more years to implement.

It's worth noting that Bevy's RFC usage was already in decline before these procedural changes, and these lackluster RFC success numbers looked more or less the same a year ago.

One thing I was not expecting was the degree to which HackMD has displaced the RFC writing process. It ended up being essential to the working group process and has led to a substantial increase in the amount of writing done by the community as a whole. Our HackMD org is now filled with content that would fit very nicely under the criteria of RFC 3. Much of it is loosely structured or incomplete. But timely information, even if partial, is preferable to none. And incomplete work is, in this case, an invitation to collaborate.

I think HackMD has largely freed us from the friction, formality, and stuffiness of RFCs. As a result, the community has written a lot of really great stuff! The org now has 59 members (and 25 non-member contributors) and has produced 72 notes. That's close to 74 RFCs Bevy received over the last five years. So in that regard, I'm incredibly pleased.

The new process has met many of its stated goals as well:

The maintainers have said "no" to working groups that it wasn't time for.
Maintainers have been able to conclusively halt work they disagreed with.
Proposed working groups have failed to form due to lack of interest.
Many working groups incorporate significant work from multiple core contributors.
New contributors do join working groups and often make meaningful contributions.

Most importantly: the new process does seem to have reduced burnout and fostered healthy and sustained collaborations. The "Better Audio" working group, for example, has been quietly trucking along for more than a year now, with multiple brilliant contributors regularly producing amazing work, and no signs of slowing down. Is this because that working group is full of amazing and uniquely talented contributors? Yes! But I hazard it's also because it's become Bevy’s dedicated audio-nerd hang-out space. A tiny community has formed around that feature, and it keeps people motivated to work on it.

What's Still Hard

Nothing is perfect, and there are still issues with the working group process.

You can't manufacture interest. Some working groups still languish.
Rendering tends not to create working groups, and some of the rendering working groups have struggled with participation.
We potentially have started too many working groups at once, and our core contributor base may be spread too thinly between them.
Several working groups have proceeded directly to implementation, without completing their design documents or getting them approved.

While the working group process feels less formal and heavy-weight than RFCs, it also has a tendency to let boundaries blur. Every group needs at least three active participants to sustain it, but it's hard to say exactly what constitutes "active", or how many groups a single person can reasonably be "active" in at once. The line between the "design phase" and "implementation phase" is also pretty permeable.

There's also a degree of permeability between what is and isn't a working group. When I got Charlotte's opinion on the scarcity of rendering working groups, she pointed out the rendering contributors

[...] tend to be pretty well aligned relative to other teams and do a good job building consensus in a more ad-hoc manner" without needing to actually create formal groups for each initiative. [...] It's more about forming small groups in order to execute on some sub-goal rather than generating buy-in through a higher level planning process.

This is a good take, and an indication that the formal "working groups process" is still too formal and bureaucratic for some situations. The rendering contributors are extremely well aligned and highly productive, so clearly whatever they are doing is working. Still, I'd still like to encourage them to write more stuff down, and to make their informal groups easier to discover and join. Perhaps there are ways we can change the process to make it work better for them.

Maybe Try It?

To summarize, we took an existing semi-dysfunctional RFC process and

Shifted the emphasis from the documents themselves to the communities that write them,
Gave those communities names, gentle oversight, and spaces to communicate,
Made it much easier for them to write and edit documents collaboratively.

People really seem to like the new process; more stuff has been getting written down, more newcomers are able to help with big initiatives, and many complex features are proceeding swiftly from design and into implementation.

It has been amazing to see the Bevy community come together over the past year for these 27 different initiatives. I cannot wait for the day when — some time in the next year — the first editor working group is officially launched. I'd also be eager to see other open source projects experiment with ways to get closer to the "old school" spirit outlined in RFC 3.