Hector Martin – [Patch] Maintainers: Remove Myself
lkml.orgThis whole thing is, at it's core, an architectural question. It's quite simple: How much should the C code have to change to allow better Rust support?
Despite his position as "Benevolent dictator for life", Linus doesn't really have the chutzpah to stand up and say "we're doing it this way"[1], so these sorts of questions try to get resolved through petty bickering and namecalling instead (which is the opposite of consensus-building and healthy debate and just makes people more entrenched and unwilling to compromise).
[1] - He was already ousted from the project once, I don't think he wants to try it again.
That question as already answered when the experiment was greenlit: it doesn’t have to at all.
That question is also not what is at the heart of this conflict. It’s someone that categorically rejects the experiment, even though the project has decided to give it a try. It’s entirely social and not technical.
The reason is entirely technical and I think it has been formulated clearly - maintaining cross-language codebase, or one of the most important kernel modules FWIW in this case, certainly does not make things simpler but only more complex - communication-wise, time-wise, resource-wise, uncertainty-wise etc. I think this is pretty clear and not really debatable IMO.
The actual question is if that overhead is worth it and it seems that there is no clear answer on that question despite many Rust-force claiming otherwise. I can totally understand the position of kernel maintainers.
What I cannot understand is the following shameful, almost unbelievable, act from Hector Martin
> If shaming on social media does not work, then tell me what does, because I'm out of ideas.
If there's a single most reason why Rust-in-Linux will fail it is going to be because of the immaturity and entitlement of individuals in Rust community.
> The reason is entirely technical and I think it has been formulated clearly [..]
He clearly expressed a technical opinion based on his own beliefs, and that’s all. There was no reasoning.
He did not even acknowledge the fact that it is going to be maintained separately from the actual generic kernel code (rust/kernel/dma vs kernel/dma).
Anyone can easily formulate a sentence that seems coherent and correct, but it can be proven completely false in 15 seconds with actual data.
IOW: just because someone calls it a technical argument it doesn’t make it one.
This is a matter of opinion - specifically, the opinion of a single person.
> If there's a single most reason why Rust-in-Linux will fail it is going to be because of the immaturity and entitlement of individuals in Rust community.
Indeed there was immaturity from several individuals.
One question: if you act immaturely towards someone and they react immaturely, who's fault is it? The person who reacted or you?
I do not believe there is a widespread issue of entitlement: if you follow the discussions on the ML and observe how the R4L project has progressed so far, the only "entitlement" that individuals in the R4L project may seem to have in common is the desire to be treated with respect and for discussions to focus on technical arguments.
For me, this is the bare minimum to be expected.
That technical decision has already been made, by Linus: it is acceptable for the purposes of this experiment, and possibly, for good. So objecting on those grounds is a project management issue.
It has certainly not been made simply because of the fact that Linus is not a boss to the people who are contributing or maintaining the code. There was however a decision to have Rust as an experiment in the kernel but not such that it will bend and change the rules and put a technical burden on everyone else. This issue is solely a technical burden and not a good guy vs bad guy, as many Rust developers would like it to appear by cherry picking the comments from the context.
I think you missed the Linus reply on the topic
> How about you accept the fact that maybe the problem is you.
> You think you know better. But the current process works.
> It has problems, but problems are a fact of life. There is no perfect
Linus is not a boss, that's correct, but he is the BDFL. He doesn't always choose to make every decision himself, but the decision to allow Rust in the tree was made by him.
> not such that it will bend and change the rules and put a technical burden on everyone else.
Absolutely. That is very important. We are in agreement here.
> This issue is solely a technical burden
Here are the entirety of the arguments made:
"No rust code in kernel/dma, please" when no Rust code was being added to kernel/dma.
"Keep the wrappers [duplicated in every driver] instead of making life painful for others." and "interfaces to the DMA API should stay in readable C code". He is not responsible for those wrappers, nor for fixing them when the underling APIs change. This is an argument against the existence of a Rust API for DMA in general, on the conceptual grounds that it exists at all, not a technical argument.
"Maintaining multi-language projects is a pain I have no interest in dealing with. If you want to use something that's not C, be that assembly or rust you write to C interfaces and deal with the impedence mismatch yourself as far as I'm concerned." They have already agreed to this.
"If you want to make Linux impossible to maintain due to a cross-language codebase do that in your driver" it was already decided that a cross-language codebase is acceptable in general. That is, it is on technical grounds (though poorly argued), but since a conclusion was come to already, it's no longer a technical argument, it's disagreeing with the process itself.
"The common ground is that I have absolutely no interest in helping to spread a multi-language code base. I absolutely support using Rust in new codebase, but I do not at all in Linux." This is a non-technical argument.
That's it. All of the technical aspects were agreed to. The only other arguments are either about the process in general, or ideological.
> not a good guy vs bad guy,
I do not think He is a bad guy. I think he is being obstructionist on ideological grounds, and does not bring anything new to the table.
> I think you missed the Linus reply on the topic
In this case, Linus is talking to Hector, who is not involved in the patch, nor as far as I know a member of Rust for Linux, though he does support their aims. He purely replied to the social media stuff, and made no comments on the patch or the arguments about it.
So do you disagree with the fact that merging and testing the code will be much more difficult or you simply choose to ignore it for the sakes of idealogical wars?
I think that statement is too broad to be useful. To make a decision like this, you need to actually argue the real pros and cons of doing so, rather than just saying "it's much more difficult."
All of this was brought up and discussed thoroughly when the experiment was greenlit, and any difficulties created by including Rust, "much more" or no, was deemed an acceptable tradeoff for the benefits of doing so. That's engineering.
Sure, engineering is all about the tradeoffs, there's no black and white.
So, what is an advantage of mixing two different languages over one kernel module, in this example DMA? What problem does introducing another language solve and what would be the alternatives?
> So, what is an advantage of mixing two different languages over one kernel module,
This is not what is happening. What is happening is the C module is getting a Rust wrapper, to make Rust code be able to call into it more easily. This is because Rust is currently being used for drivers, and DMA is very useful for drivers, so they need to call into the C code somehow.
> What problem does introducing another language solve and what would be the alternatives?
The ones Linus found persuasive are mostly that Rust's memory safety can bring more stability and security to the kernel, and that Rust is likely to attract new developers. Obviously that's just some of it, but those are the main technical and social benefits from including Rust.
The alterative is to not do it.
> What is happening is the C module is getting a Rust wrapper,
Yes, I understand that. I read the argument on the mailing list. The issue is that changing the DMA code will in turn potentially break the drivers written in Rust and the maintainer has no incentive to do that extra mile himself to make this not to happen. Rust devs OTOH offer say "we're gonna help you fix it" but this obviously creates an extra technical and communication burden. This also means that the future might look like "xy DMA patch didn't manage to get into this release because we didn't manage to fix all the zy Rust dependencies".
To make this work swiftly every kernel developer would need to become very comfortable with at least two toolchains and two very different programming languages. Otherwise, it's simply too complicated and it's a real issue that I do not see as ideological. I think this is a big ask.
> The ones Linus found persuasive are mostly that Rust's memory safety can bring more stability and security to the kernel
This is the tricky part - apart from the theory, we don't actually know that. We don't know if the immediate effects of kernel code written in Rust would result with X times less segmentation-faults or CVEs. This is a hypothesis that is especially tricky to be proven in the kernel-level code, a code that is destined to be using a lot of unsafe blocks.
Even if that hypothesis would have been somehow proven, from an engineering PoV, the pain that it aims to solve still needs to be quantified in order to assess if the change is worth it or not. I don't have that number but I honestly don't think that kernel developers spend too much of their time and resources fixing the memory-related bugs. If they did, then there would be a pretty convincing and strong incentive for them to switch, wouldn't it?
So, when we put the dogmatic arguments aside and put those few (big and important) things into a perspective, I think it is not unrealistic to have people oppose to the idea.
> The issue is that changing the DMA code will in turn potentially break the drivers written in Rust and the maintainer has no incentive to do that extra mile himself to make this not to happen.
He has no reason to do anything to make it not happen. He is 100% within his rights to ignore things entirely.
> Rust devs OTOH offer say "we're gonna help you fix it" but this obviously creates an extra technical and communication burden.
No, they say "we are going to fix it." He has no responsibility for Rust code. This is the terms of the experiment, which has been clear for years now.
> "xy DMA patch didn't manage to get into this release because we didn't manage to fix all the zy Rust dependencies".
This is not how the process works. All DMA patches get in, if the Rust is broken, it would be "The Rust code that depends on DMA did not get in this release."
I don't think this is a fair characterization. I think the accusations from the Rust side that the project is "sabotaged" or "non-technical arguments" etc. are a typical symptom of denial. All the hype and enthusiasm related to Rust is that you can just write something easily in C instead of Rust (or rewrite) and then all problems related to safety are solved. And some of those people really believe this nonsense. The real world complexities unsurprisingly do not go away simply by using Rust. The signs are all over: The project is moving very slowly, the very enthusiastic people who move it along with sheer will power get burned out, there also CVEs in Rust code, etc. But if you bought all the Rust arguments and are now invested, it is difficult to accept that the solution of the safety problems is not as simple "rewrite to Rust". So the more immature part of the Rust communities starts to blame politics. Now, I do not want to say that there are not grains of truth in all these points (I do think memory safety is great, I do think that C ecosystem has issues, I agree that there are many good ideas in Rust, and certainly skepticism from others makes it harder to push something forward in a community, etc.), but overall I think it is mostly delusional ideas from Rust folks encountering the real world, and instead of critical self-reflection on what needs to improved both in terms of social aspects of collaboration and maybe also on the technical side, they blame others for sabotaging the project and organize a social media circus to force their ideas on a community (which of course, is super toxic and evil)
I don't entirely disagree with some of your points, however, they are not what the recent discussion has been about. Plainly, a maintainer has unilaterally rejected the addition of rust code to assist with DMA and requested that code be duplicated in every driver (which also ignores the fact that the patch never added rust code to kernel/dma to begin with). It strikes many as strange that the experimental addition of rust-based drivers (greenlit by Linus orignally) has come to a head in this way:
"The common ground is that I have absolutely no interest in helping to spread a multi-language code base. I absolutely support using Rust in new codebase, but I do not at all in Linux."
I think rejecting code in his area is his right as a maintainer of this area (to the extend this is the case, I haven't checked). Also the patch adds a file kernel/dma.rs so I am confused about your comment. I also happen to agree that maintaining multi-language code is a pain and I understand that he does not want this imposed on him. This may make it harder for Rust kernel developers, but I can not see how this is "sabotage of the project".
> (to the extend this is the case, I haven't checked)
It's not his area.
> Also the patch adds a file kernel/dma.rs so I am confused about your comment.
It adds rust/kernel/dma.rs, not kernel/dma. That is, it adds that file here: https://github.com/torvalds/linux/tree/master/rust/kernel not here: https://github.com/torvalds/linux/tree/master/kernel/dma
If it is not his area, he should have no power to stop it anyway, so it is even more strange to call it "sabotage". Or has any kernel developer the right to nack anything? This seems unlikely to me. So then it is just some disagreement.
You have to remember that Linux development isn't done like most open source projects where there's one upstream tree everyone sends patches to. Linus pulls in whatever code he wants. A nack means that Hellwig won't pull it into his tree, but that doesn't mean it can't go in someone else's, and end up upstream anyway. The only reason he was even cc'd on the patch is because he's the relevant subsystem maintainer being wrapped, as a courtesy.
As I said, the idea that this is then "sabotage" is completely ridiculous and just shows how toxic this maintainer was that now removed himself.
There are multiple comments that meet the exact definition of sabotage. If this:
"You might not like my answer, but I will do everything I can do to stop this."
is not intent to sabotage (even if it might not be successful as Linus could pull in the patch anyway), then what possibly could be?
Pointing out the ridiculousness of comments like this and suggesting the R4L folks push forward while ignoring them doesn't scream toxicity. Refusing to compromise with the R4L devs and calling the additions a 'cancer' has expectedly caused a stir.
I think you need to look up "sabotage" in the dictionary.
sabotage /ˈsabətɑː(d)ʒ/ verb
1. deliberately destroy, damage, or *obstruct* (something), especially for political or military advantage.
2. to intentionally prevent the success of a plan or action.
Definitions from Oxford, Collins, and Cambridge all fit the bill. Even dictionary.com has "any undermining of a cause."
Yes, and now compare to what happened. You could differentiate between words and actions and what exactly was affected. For someone to "sabotage the Rust experiment in the kernel" you would need to determinate that that person did something that a) effectively damaged / obstructed the project (which - if GP is right about that that person has no power to stop the merging of the patch anyway - is a dubious claim), etc.
Misrepresenting the voicing of opposition to some process as "sabotage" seems completely out of line for a any kind of community project. If you define things so loosely, then every side in any disagreement could always label the other side of doing "sabotage". This reflects the sentiment of many Rust people to "be on the right side of history" where everybody else automatically is wrong and even voicing objects and criticism is already "sabotaging" on the true path.
If you don't think what Hellwig did obstructed the project, then I guess we fundamentally disagree. Again, the R4L folks could work around this by getting the code pulled in by Linus, but that doesn't stop the fact that a senior maintainer has made it explicit that they will do everything in their power to stop this.
Code that wasn't his to reject was NACKed, causing a large amount of uncertainty about how to proceed with drivers that use DMA, and around the R4L project in general. At the absolute least, this is plain intent to sabotage (but IMO it is clearly more than intent at this stage). The core of what you are saying is that this has/will have absolutely no impact on anything to do with future R4L progress. The explosion of discussions around this exact topic across various forums with abundant disagreement from maintainers and R4L folks running counter to that idea are irrelevant I guess.
I'm not even a "rust person" and nobody has said anything about "being on the right side of history" except you. If that's how you see this discussion then we're not going to get anywhere. I wish you well, and urge you to in future engage in good faith and consider that not everybody is some boogeyman "on the true path" evangelist.
A senior maintainer has a different opinion and expresses it. Your point seems to be that because he is not on board with the plan and expresses this, this is already "sabotage". Of course it makes things harder if not everybody agrees. But this is not the same thing that people disagreeing with your plan and say so do "sabotage". Sorry, this is ridiculous.
There is a large difference between "I do not think this is a good idea" vs "do not do this", in particular given the position Hellwig has in the kernel as a listed maintainer of the DMA mapping helpers.
No single technical reason was given besides a non-specific opinion on the "messiness" of multi-language projects.
> I think the accusations from the Rust side that the project is "sabotaged" or "non-technical arguments" etc. are a typical symptom of denial.
When someone says
> You might not like my answer, but I will do everything I can do to stop this.
Against the decision that the project made, that is very straightforwardly sabotage.
> All the hype and enthusiasm related to Rust is that you can just write something easily in C instead of Rust (or rewrite) and then all problems related to safety are solved.
You cannot accuse others of being hyperbolic and then be so yourself. No serious person claims that it’s easy or that it solves all issues.
I don’t really even want to engage with this rest of this post, honestly it says more about you than about Rust.
Well, people definitely often claim that Rust solves all memory safety issues (not true) and they also often neglect to carefully distinguish between memory safety and other safety issues (e.g. already in terminology "safe language").
> That question as already answered when the experiment was greenlit: it doesn’t have to at all
Where? And in light of the ongoing infighting and lack of clear parameters, do you think that was truly sufficient?
When the Rust for Linux experiment was approved.
It was sufficient, but that doesn’t prevent people from acting in bad faith. Which is what is happening here.
It may be technically insufficient, if the RfL folks can't integrate Rust without C changes. And if that's true, (and it seems to be true) then it's time for an architectural discussion to come to an agreement on if Rust for Linux should be pursued or if C code flexibility and readability is more important.
> if the RfL folks can't integrate Rust without C changes
The RfL folks can integrate Rust without C changes.
All of the changes in the patch (ever since v1) have been entirely contained inside the rust directory, with most of the changes being in the rust/kernel/dma.rs file. There were no changes to C files, or to files inside the kernel/dma directory.
It is what it is... However if you want volunteers to contribute to the kernel and introduce rust as a way to get more people to join this looks pretty bad... I'm sure stuff will figure itself out in the long run it's just sad that the discourse exists at all. I guess the linux kernel will never transition to memory safety via a new language? Idk what else can be done.
Well, I very much hope that the Linux kernel will never transition to Rust. The language is far too complex for my taste. IMHO Rust people should just build their own kernels and if those are better they can replace Linux just like the C++ microkernels did.
The complaints I have with rust are going to be on the async side. We need memory safety in the kernel, it dosen't have to be majority but having vital parts in rust could be a game changer. The kernel must evolve, all players are moving this way. Swift is being introduced into Darwin, Rust into Windows. Memory safety is not a fad and the only tangible options right now is rust. Swift could be interesting but creates other problems, zig is not 100% safe, and carbon is vapor ware. Could there be something better than rust? Yes. Do I think rust can be annoying at times? Absolutely... But the way I look at it, we don't have another shot.
I think a far better approach to memory safety in C projects is to evolve tools and annotate existing code. It is also not true that everybody moves to Rust and I think the claims about memory safety are overblown (due use of unsafe Rust). In my experience as a observer and contributor of open-source and free software for the last decades, these changes driven by enthusiastic and opinionated small group that try change how existing projects work are very dangerous and often contraproductive both in terms of technical and social risks for the projects. It is far better to start alternative projects that compete on their own merits and if the techniques are truly superior, then those will eventually be competitive. Going into existing projects and changing them according to "opinions" is a really bad idea.
> I think a far better approach to memory safety in C projects is to evolve tools and annotate existing code.
The advantage of Rust is that it already exists. Someone has to do the work to bring that to fruition before it's practically applicable.
> these changes driven by enthusiastic and opinionated small group
In this case, that group includes Linus.
> Going into existing projects and changing them according to "opinions" is a really bad idea.
The Rust for Linux folks already worked on Linux for years. They aren't outsiders.
I agree that memory safety gets over-hyped and is often treated like a silver bullet solution. However, I do believe the problems of memory unsafely to be real and to be a threat (just less than what the popular opinion is). We've tried using static analyzers and C and AI driven detection to limited success, we cannot guarantee correct C code, this is just a hard problem when a project as big and important as the kernel exists.
Rust was an experiment proposed by Linus so I don't really see this as driven by a small group, it should be the standard as that's what Linus wanted to try. It's an experiment that's having social problems.
I think alternative projects are a good idea and some are already being and have been created, but the Goal of this experiment was to help Linux evolve in a direction to keep the kernel competitive and healthy.
I think Rust is a great idea but it might not be the right fit for the kernel socially, which is unfortunate. Maybe the only option forward here is improving C... but that seems like an impossible task without breaking changes. I do hope for the best with the kernel and with the RFL maintianers.
There are definitely people using static analysis to write memory safe C. This is not really practical though, but getting a reasonable level of memory safety is very much possible using tools and good design. People dismiss this because it is not perfect, but this completely fails to take into account that - in practice - Rust is also far from being memory safe due to use of unsafe.
Whether it was proposed by Linus or not, it still driven by a small amount of people. I do not think it was a good idea from Linus to allow it. I understand the arguments he had, but I think it was a mistake. In the past, he kept the C++ programmers out and I always thought this was a wise decision. And the arguments for C++ were not entirely different, except it was not memory safety but object orientation which was "clearly needed" and "the right way" and everything C is old, obsolete, and wrong. By just saying no he kept this discussions away.
Rust is memory safe by every definition. Unsafe is an escape hatch that is only used when safe won’t work and assumptions can be made by the programmer. This is a needed solution and does not take away from rust as a safe language.
To me Linus is the leader so if he wants to try it we should try it. If he wants it gone then it should be removed all the same. Linus did not include rust for something like a programming paradigm it’s for the safety of the kernel and hope to attract more younger contributors.
The kernel will eventually need to evolve and I think this experiments are extremely important to prevent the kernel from turning irrelevant.
Safe Rust is memory safe when used exclusively while unsafe Rust is obviously not memory safe, and a mix might also not be. If you use unsafe blocks carefully in Rust it is still quite memory safe but if you use unsafe blocks carelessly is not very memory safe. My favorite example is the CVE in a matrix transpose function: https://github.com/advisories/GHSA-5gmm-6m36-r7jh A similar error in C would be taken by Rust proponents as clear sign how dangerous C is. Overall, in practice, the safety of Rust is not so clear as proponents claim, except when strictly sticking to the safe part.
I think the experiment is a mistake and the kernel would obviously stay relevant even without this. But I am happy to let Linus make this decision. I just think it is a poor decision.
I agree, unsafe rust is only meant to be used when absolutely needed. It’s up to the programmer. Frivolous use of unsafe should be rejected from any respectable code base.
I didn’t mean to say that without it the kernel would be irrelevant. I meant more that this is just one possibility for the future of the kernel and I think it’s important that we try as it has to always be better. Given the response from current maintainers I also question if the experiment should continue.
This is a remarkably disappointing end to Hellwig's original NACK, yet entirely expected based on the recent treatment of R4L devs.
Another for the pile of reverse-engineered devicetrees that started downstream, tried to negotiate with upstream and ended up realizing the futility of it all. Godspeed, may your path to depreciation be slow and painless.
I hope they keep up the good fight …
The good fight was already lost when Apple didn't lift a finger to help. No documentation, no driver code, just an open-enough iBoot interface to flash some code onto. Valiant programmers can take a hack like that pretty far, but at some point they have to ask what they're fighting for. Is Asahi interested in adapting to the norms of the Linux kernel, or do they insist on bending the Linux kernel to meet them?
I understand their exhaustion, but aiming their frustrations at Linus for having an entirely level-headed response shows that they can't read the room. This is the Linux project, you're going to have problems merging enormous codebases that reverse-engineer poorly understood hardware. I support Rust in the Linux kernel, but this is the kind of envoy that will spoil the effort.
> I understand their exhaustion, but aiming their frustrations at Linus for having an entirely level-headed response shows that they can't read the room.
I agree that Linus's response wasn't over the top, but Hector didn't quit because of that one email. Heck, as far as I know he wasn't even involved in writing the patch in question. The Asahi folks have had a number of issues dealing with other maintainers for at least the past couple of years. This is a "the straw that broke the camel's back" situation.
The Apple hardware is compelling enough that I think they will keep it up. We are funding them — at least I am trying to
If I poured out a beer for every "compelling" chipset depreciated by the mainline Linux kernel, I'd be going out to buy 6-packs on a weekly basis.
It's a neat initiative, but you should have been preparing for this result since the beginning. No official support means no official support.
What chipsets were deprecated that are still mainstream?