Bcachefs goes to 'externally maintained'

> Surprising, can you please share some commit IDs?

Try git log v6.16-rc1..v6.16 -- fs/xfs

> The most important points seem to be missing from that list: size and nature of the changes. For both risk and maintainer bandwidth reasons.

> If a "critical bug fix" has a non-negligible risk of regression, then either there's a clear divergence on the definition of a "critical bug fix", or the whole feature should be temporarily disabled (cause it has no bug fix simple enough for an RC phase). Or just filed and advertised, e.g. "don't use version X".

There was ~0 risk of regression with the patch in question.

bcachefs's journalling is drastically simpler than ext4's: we journal btree updates and nothing else - it's just a list of keys. For normal journal replay, we just sort all the keys in the journal and keep the newest when there's a duplicate. For journal_rewind, all we do is tweak the sort function if it's a non-alloc leaf node key. (We can't rewind the interior node updates and we don't need to, which means alloc info will be inconsistent; that's fine, we just force a fsck).

IOW: algorithmically this is very simple stuff, which means it's very testable, and it's in one of the codepaths best covered by automated tests - and it's all behind a new option, so it has zero affect on existing operation. This is about as low regression risk as it gets, and the new code has performed flawlessly every time we've used it.

> - Either a significant number of bcachefs people use Linus' mainline and trust it with their data. Then that branch is not really "experimental" any more (whatever the label says), and no large change should ever be submitted in the RC phase but only small, "critical bug fixes"

No, you've got it backwards. The experimental label is for communication to users, it's not for driving development policy.

We ALWAYS develop in the safest way we practically can, but we do have to balance that with shipping and getting it done. Getting too conservative about safety paralyzes the development process, and if we slow down to the point that we're not able to close out bugs users are hitting in a reasonable timeframe or ship features users need (an important consideration when e.g. we've got a lot of users waiting for erasure coding to land so they can get onto something more manageable, robust and better supported), then we're not doing it right.

OTOH, there's generally no need to hair split over this, because if you're doing things right, good techniques for ensuring reliability and avoiding regressions are just no brainers that let you both ship more reliable code and move faster: if you strike a good balance, most of the techniques you use are just plain win/win.

E.g. good automated testing is a _massive_ productivity boost; you find bugs quicker (hours instead of weeks) while the code is in your head. Investing in that is a total no brainer. Switching from C to Rust is another obvious win/win (and god I wish bcachefs was already written in Rust).

Work smarter, not harder.

But one of the key things we balance in "fast vs. safe" is regression risk, and that does vary over the lifecycle of a project. Early on, you do need to move quicker: you have lots of bugs to close out, features that may require some rearchitecting, so accepting some risk of regression is totally fine and reasonable as long as those regressions are minor and infrequent compared to the rest of the bugs you're closing out (you want the total bugcount to be going down fast) and you're not creating problems for yourself down the road or your users: users will be fine with that as long as you're quickly closing out the actual issues they hit. I eyeball the ratio of regression fixes to other bugfixes (as well as time spent) to track this; suffice it to say regressions have not generally been a problem. (The two big ones that we were bit by in the 6.16 cycle were pretty exceptional and caused by partly by factors outside of our control, and both were addressed on multiple levels - new hardening, new tests - to ensure bugs like that don't happen again).

The other key thing you're missing is: it's a filesystem, and people test filesystems by using them and putting their data on them.

It is _critical_ that we get lots of real world testing before lifting the experimental label, and that means people are going to be using it and trusting it like any other filesystem, and that means we have to be supporting it like any other filesystem. "No big changes" is far too simple a rule to ever work - experimental or not. Like I said earlier, you're always balancing regression risk vs. how much users need it, with the goal being ensuring that users have working machines.

There's also the minor but important detail that lots of users are using bcachefs explicitly because they've been burned by another COW filesystem that will go unnamed (as in, losing entire filesystems multiple times), so they're using bcachefs because even at this still slightly rough and early state, the things they have to put up with are way better than losing more filesystems.

That is, they're using bcachefs precisely because of things like this: when something breaks, I make sure it gets fixed and they get their data back. Ensuring users do not lose data is always the top priority. It's exactly the same as the kernel's rule about "do not break userspace". The kernel's only function is to run all the other applications that users actually want to run: if we're breaking them, we're failing at our core function. A filesystem that loses data is failing at its core function, and should be discarded for something better.

> That sounds like 20 years of filesystem experience and 0 year experience of not being the boss?

Well, if everything comes down to authority and chains of command now then maybe kernel culture is too far gone for filesystem work to be done here. Because that's not good engineering: good engineering requires an inquisitive, open culture where we listen and defer to the experts in their field, where we all learn from and teach each other, and when there's a conflict or disagreement we hash it out and figure out what the right answer is based on our shared goals (i.e. delivering working code).

> Maintainers don't really have time to explain;

That's a poor excuse for "I don't have time to be a good manager/engineer".

In engineering, we always have to be able to justify our decisionmaking. I have to be able to explain what I'm doing and why to my users, or they won't trust my code. I have to be able to explain what I'm doing and why to the developers I work with on the bcachefs codebase, or they'll never learn how things work - plus, I do make mistakes, and if you can't explain your reasoning that's a very big clue that you might be mistaken.