Buffer overruns, license violations, and bad code: FreeBSD 13’s close call
arstechnica.comWhen I was younger and had more time, I loved the BSDs. FreeBSD seemed so coherent, I loved the ports collections etc. And FBSD had a reputation for good code. From 2000-2004 I ran FBSD exclusively on my servers. Then slowly, FBSD seemed to start to splinter. First Matt Dillon took off for DragonflyBSD and then I started using Linux more. When I needed a firewall, I chose OpenBSD because Theo seemed to have a tight (some would say too tight) grip on the project.
When FBSD integrated ZFS, I took a look and decided that while I love that file/storage system, FBSD itself had turned more into a lesser version of itself. Perhaps this was due to more pressure from Linux, and fewer developers/contributors.
This entire Wireguard debacle has pretty much turned me off ever using FBSD again. From the inclusion of Sendmail as the default MTA (really? over Postfix) to the lack of development control outlined in this article, I can't trust it.
Perhaps Theo's strategy was the better path.
You also have NetBSD. WireGuard is going to be available with release 10.0 and the default MTA is Postfix. NetBSD is a low profile BSD, but very capable.
netbsd also has the biggest, most cruftly code base and least developers. fwiw
This is an incredibly good piece of journalism. It gives tangible examples of the issues found, puts it in context of prior work done by the developer, features first-hand verification of the claims made against the code; the author reached out for comment to all the parties involved.
I'm also consistently impressed by the quality of comments at Ars Technica whenever I visit the site.
This convinced me to subscribe. We need more journalism of comparable quality.
This does not paint FreeBSD in a good light.
“you either have a commit bit (enabling you to commit code to FreeBSD's repositories) or you don't. It's hard to find code reviews, and there generally isn't a fixed process ensuring that vitally important code gets reviewed prior to inclusion. This system thus relies heavily on the ability and collegiality of individual code creators.”
From my perspective, this whole thing is due to a severe failure of the development process. The sub-standard code should never have been committed. But if there is no process, is it really a failure? Or is this just how it is on FreeBSD?
> Or is this just how it is on FreeBSD?
It's not. We do a lot of code review, and it's done publicly. It's easy to look at the commit logs. I find it telling that the article doesn't spend even one word trying to delve into our code review practices. This case was an aberration.
The core group had a chance to state this when they were asked for a response. This one word could’ve come from any of the FreeBSD devs contacted by the author. Instead back came either nothing or a general statement without any real message.
That's fair. At the time, though, it wasn't clear that the author's angle was that FreeBSD doesn't have a culture of doing code reviews. We do, and I don't think one has to look very hard to see that.
The FreeBSD organization should not have had to be told what issues this incident raises, so this response raises one more flag about how the organization is handling it.
This defensiveness in FreeBSD's response, this reaction of minimizing rather than facing up to the seriousness of the problem revealed here, the attempt to redirect the conversation towards anything but the specific issue, only reinforces the impression that FreeBSD may not be ready to deal with it effectively.
Of course, as FreeBSD is open source, its users are in no position to demand anything, but any potential user may, and should, attempt to determine how likely it is meet her needs in all respects.
Well, at least once code was committed without being thoroughly reviewed. Clearly FreeBSD’s development process needs some refinement to ensure that the entire code is actually reviewed during a code review.
As a FreeBSD user, that's good to hear :)
Has there been any discussion about why the process failed in this case, and what is going to be done to make sure something similar doesn't happen in the future?
This is the review for the original commit, in case anyone is interested: https://reviews.freebsd.org/D26137
There's some discussion happening now and I do expect to see some process changes coming out of this. It's tricky. The review you link does nominally follow the process of creating a review and having some discussion, but there isn't much actual code review happening there.
Ars seems to be calling all open source software insecure. I’m not saying they are wrong but what’s the value in their article? Is it gotcha journalism, or are they warning us not to trust bsd based systems in general. The article starts as a gotcha piece but concludes by saying there’s no review in place to catch these problems.
This is just incorrect. Nowhere in the article does the author, Jim Salter, generalize about all open source software. When someone in the comments did make that leap, Salter pointed out that if these had been closed-source projects the bad code would have been put into production and no one would have known better.
You’re right. I misread the original to be general criticism of open source, but the author made his statement specific to FreeBSD release 13.
“Neither Netgate's responses, FreeBSD Core's, nor the off-record responses we heard from independent FreeBSD community members lead us to believe that there was in fact any process in place that could reasonably have been expected to catch this issue prior to it going out into the world in 13.0-RELEASE”
But it would help if he made it clear why he felt this criticism was specific to FreeBSD release13. It sounds to a naive reader like a critique that could apply to much of open source software. The author didn’t say that, but it’s a reasonable extrapolation to make.
> Ars seems to be calling all open source software insecure
> The article starts as a gotcha piece
Haha, that’s the problem. You, Netapp, value saving face so very much that you are incapable of constructive response to well-meaning, fair, and honest criticism.
Not netapp and I don’t use or endorse their products. I just don’t like angry mobs or journalism that uses angry mobs as a business model. Too much anger and retribution in the world, not enough forgiveness or compassion. If this code was up for review for months and nobody noticed the printf debugging code in the crypto then I’m sorry but blaming this one bad person seems gratuitous to me, and shortsighted. Are concludes the article by saying this is a bsd/open source problem. How does that help anyone?
Having deployed Netgate PFsense hardware at my last 3 startups words cannot express how disappointed I am in this. I have also recommended them to others. I understand that mistakes happen, but I feel their response was utter garbage. Unfortunately I am done with them and will have to find another option for the future. We need a project to put a web GUI on PF on OpenBSD (while I can sort the .conf files, not everyone can).
Thank you to Jason Donenfeld (Wireguard), Kyle Evans (FreeBSD) and Matt Dunwoodie (OpenBSD) for jumping in and fixing this in a week!
Honestly, pfSense has been a bit suspect for a while due to less than forthright development.
OpnSense is the truly open source alternative and is a solid option. It was started in protest of pfSense’s less open behavior.
Thanks. I will have a look!
A note on the article it lists as one of the code flaws "Validation functions which simply return true".
That got me thinking what's so bad about returning true? What should they be returning?
Then I realized that what article must is trying to complain about is: "Validation functions which ALWAYS return true".
Good that folks on FreeBSD have proper controls that stopped the problem before it was released, and shame on Ars Technica for bringing completely irrelevant 10+ year old eviction dispute into an article about technical issues as if it were relevant. This bullshit needs to stop. I mean I get that the guy may have some issues, and burnout is a very real thing, and if the code is low quality then it needs to be addressed, but it shouldn't be "oh and also his code is bad because of 10-year old story that has nothing to do with the code in question". We really can do without this stuff, and if they just dropped that whole section, the article would be much improved.
Calling the story "eviction dispute" is quite an understatement. They served 4 years jail sentence for sawing huge hole through tenants' living room and forging threatening letters as if they were coming from tenants.
The story is relevant, because:
1. It was already out there in the public. Skipping it completely would be unprofessional.
2. It gives a possible explanation for low code quality.
3. The developer in question is open about his past.
4. Person who went so far as resort to destruction and deceit in the physical world, could easily cut corner in the code review process:
Most important part first:
> Person who went so far as resort to destruction and deceit in the physical world, could easily cut corner in the code review process:
This is a very dangerous delusion. It implies that producing flawed code requires some kind of moral blemish, a flawed character, a bad person. Truly upstanding citizen and high moral character would never submit a buggy code, but one can't expect otherwise from a criminal.
Nothing could be further from the truth. In my decades-long experience reviewing what must be by now megabytes of code, the code quality has absolutely no relationship to the moral character. Best people can produce - and regularly do produce - very flawed code (very much including myself, of course), and even excellent coders can be busy, tired, have temporary slip of attention, be wrong about particular API or language construct, mistaken, suffer from a burnout or a work-life issue... Even at their best, people produce flawed code all the time - that's why we have code reviews in the first place! It's not to weed out "that kind of people" who try to sneak into our pristine cohorts - it's because producing good code is hard, and producing flawless code all the time is nigh impossible to do by a single person. A concerted effort of multiple smart people over time is required to achieve even imperfect, but acceptable quality - and perfection is still an unattainable goal. It's a hard work, and it can't be done alone - nothing to do with character flaws.
And yes, left to their own devices, even the best people are subject to cognitive biases and fallacies - that's why it's impossible to effectively review one's own code and you need peer review. Not because you're suspected in being a criminal or at least a sloppy coder - but because you're human and as such, your best effort will never be perfect, especially not at scales modern code is produced.
> It was already out there in the public. Skipping it completely would be unprofessional.
Nonsense. Nobody expects every article about a person to include their complete biography. People expect including relevant stuff and throwing out irrelevant one. What is unprofessional is brining in irrelevant details to smear the character of a person to prejudice the reader against him from the start (that's why this BS goes first and the substantial part goes later). Instead of trusting the reader to judge on substance, the hack first creates an emotional prejudice which would cloud the mind of the reader and make a pre-formed opinion before the substantial part ever begun.
FreeBSD shipped sendmail, for how many years? Is it actually still in ports?
Last I checked (FreeBSD 12.x), sendmail was still the default and included with the base install.
OK... With sendmail still in the base install, why is anybody talking about new security or code quality problems?
The sendmail of today is not the sendmail of the 90's.
Sure, I'd prefer postfix, but if you're just sending local email out for system checks or whatever, sendmail's okay.
As someone who is totally not aware of this: Why is everybody so horrified about sendmail being included in an OS?
Historically, there were a ton of vulnerabilities in sendmail. 1980’s C code, etc. Also, I will say its configuration format (“sendmail.cf”) is awful, though generally nobody works with it directly. FreeBSD uses “m4” to build the configs, for example.
Thank you!
LWN had a good story on this too: + [WireGuard bounces off FreeBSD—for now (LWN.net)](https://lwn.net/SubscriberLink/850098/3daef578513bff15/)
So some Twitter grand inquisitors, whose names always seem to appear if individuals are targeted, discovered some unpleasant details about someone's past.
Code quality (especially when written under pressure) is unrelated to that and I've seen horrible code from from model citizens who check all the Twitter boxes of goodness.
It seems very dangerous to contribute to open source these days if you are not in the right Twitter cliques.
The nice thing is that the FreeBSD developers who were interviewed apparently remained fair and said that the target had produced high quality code before.
> It seems very dangerous to contribute to open source these days if you are not in the right Twitter cliques.
Nope. Lots of people contribute to Open Source without being in any Twitter cliques, they're just getting on with the work and doing their best.
One could also flip what you wrote, on its head and say "It seems very difficult to be visible in open source these days if you have been doing things that are illegal or frowned upon". It's the same thing, just without the persecution complex.
At least bad opinions on Twitter are not a crime... and you know, perhaps Ars shouldn’t bring it up, but it’s hard when they themselves refer to it as a personal set back. But even though I absolutely believe everyone deserves a second chance, I really find it hard to sympathize with a person who does what is alleged, doesn’t apologize (as far as I have heard), attempts to flee the charges, then has the gull to lament over how it has negatively impacted their career. At this point it feels like they are more upset about how they had to face consequences for their actions than anything else.
I don’t wish perpetual punishment on anyone for almost any reason. But still... it feels like some necessary self-improvement is sorely missing. I certainly say this as a person who is flawed and full of anti-patterns.
Adults rarely change their essential character. Most criminals guilty of substantial predictable harm to others are unethical people. Those who integrate with society successfully learn not to do stupid destructive things. They are not likely to stop being terrible people.
Anyone who would try to drive people out of their homes by destroying them while they live in them is a terrible person without redemption.
His inability to turn away money to do work he had no intention or capability to complete just demonstrates this.
Perpetual punishment is pointless but everyone has a right to know the kind of person they are dealing with.
Best thing is to not use Twitter in the first place.