Absolutely the best outcome. BSDs has always been ( to me at least ) about getting things right, take time to get it baked before committing. The old, out of fashion style of getting things done properly and not shipping for the sake of it. Which is both a good thing and a bad thing in the modern world. But it is a trade off.
I also hope FreeBSD sort of look into why it was committed in the first place.
Hopefully this also set the tone for Scott and Netgate. Even having mild scepticism of the one sided bash to them on HN had me downvoted into oblivion.
> I also hope FreeBSD sort of look into why it was committed in the first place.
It is a widely desired feature that no one else was working on and unfortunately, did not see adequate review. FreeBSD largely trusts committers to seek out review and submit quality code. There are certainly downsides to this model, sometimes more impactful than others. Unfortunately, FreeBSD does not have the large body of employer-paid maintainers Linux has available to thoroughly code review every patch that lands.
WireGuard is gone from the kernel in 13.0-RELEASE. Given the choice between "buggy" and "less than a week old", we're going with the third option of "you can ship a kernel module via the ports tree".
They are removing both implementations(the new and the broken one) in order to put more work and review on the new one, and release it properly at a later time.
The dramas [0] between PFSense, OPNsense, and IPFire [1] always seems to come up.
I ended up going with PFSense and it works fine. It's open enough that you can always dive in to figure out what's going on. Perhaps philosophically suboptimal, but for all practical purposes it's worked great for my home!
It’s a fairly recent turn of events. First it was difficult to compile on 2.4 because three left out closed source dependencies that their build scripts relied on.
With 2.6 they are basically diverging entirely. Albeit they are still trying to argue they are foss.
The issue I have is if I’m going with an edge security appliance that has code that can’t be easily audited by security pros better than me, I’ll go with Pali Alto or Cisco who has entire branches and teams dedicated to security like Talos/snort. They are less succeptible to security errors and have a customer base that straight affects national security. So even Alphabet agencies will report exploits and 0days to them.
With their wire guard shenanigans it’s clear they are a small team and closing off the code base means I’m now relying on people that act this way to criticisms for security. I don’t really care about internet drama and it’s a reason I’ve stayed with pfsense to now. But pragmatically their choices mean I have to change. Which is okay too.
The shade I occasionally see thrown toward pfSense is curious to me. This isn't push-back at the parent comment but me expressing a bit of confusion.
I've used pfSense since 2009 or so. I was skeptical when Netgate entered the picture but since I've had no reason to complain. It's been a continuous and usually smooth timeline of serving me well.
A relevant sidebar is that I've been part of different, stellar volunteer efforts - started by a core team that was trying to improve or fix something worthwhile. It is inevitable that core teams members will eventually run low on time/energy and changes must follow. Those changes can be anything and usually are.
> The shade I occasionally see thrown toward pfSense is curious to me.
Every last bit of it is deserved. They made a promise to keep pfSense open source and they broke it as soon as they could. I see them hiding behind it's the newly announced pfSense Plus that is closed source, not pfSense CE and it's pure weaseling.
I still use pfSense but I feel bad for ever being excited about it and contributing to their popularity.
I'm not sure that over 10 years later is "as soon as they could". NetGate has made a huge number of open source releases, and while they have not held exactly to the platonic ideal of open source (literally every bit on the disc comes from an open repo) I think we can all agree that the vast majority of the existing CE code remains open. I also think that they get a lot of shade because some of their developers have been some of the loudest jerks in open source.
In my opinion, at the moment we have Schrodinger's open source: in the box there's a future pfSense CE which is well-maintained but differentiated from their commercial offering of pfSense Plus, and there's a pfSense CE which languishes from a lack of new features and slowly accrues an ever-larger trail of closed-won't-fix bugs.
At this time, which future will develop is anyone's guess; I suspect even NetGate don't really know. Even if they're planning on effectively abandoning CE in place, a backlash in the community could cause that to reverse.
> At this time, which future will develop is anyone's guess; I suspect even NetGate don't really know. Even if they're planning on effectively abandoning CE in place, a backlash in the community could cause that to reverse.
It seems like a certainty that users will shift over to the free version of pfSense Plus for the eventual performance advantages, if not for the REST API alone, and then pfSense CE will slowly wither. We'll see, but I really think you're being overly optimistic entertaining an alternative scenario :)
> However, you are directing your disdain (about pfSense) toward us.
I don't think I am; who's us in that sentence?
> To what end? What is it you want to achieve?
I'm scratching an itch. If Netgate can screw the community that helped pfSense gain popularity then surely it is perfectly acceptable for a member of that community to express a little disdain.
> it is perfectly acceptable for a member of that community to express a little disdain.
Okay. I never inferred otherwise. If venting is the total of your goal here are you okay we blow that off or is there something else you're hoping for?
To be clear, I've no animosity toward your posts. My 'hidden' agenda is this: Because hostility takes a toll on the recipients (us), I'm curious if what you're getting in return is worth it.
> “Because hostility takes a toll on the recipients (us), I'm curious if what you're getting in return is worth it.”
We aren’t the recipients of the hostility; Netgate is. I feel no hostility directed towards me when reading anfogoat’s post. In fact, I thank them for openly expressing their disdain towards Netgate here, as it gives others like me more information to look into and come to our own conclusions on.
> To be clear, I've no animosity toward your posts.
No worries, no animosity assumed.
> If venting is the total of your goal here are you okay we blow that off or is there something else you're hoping for?
I don't like venting. I said I was scratching an itch but venting makes it sound like it had no substance at all and suggests what Netgate did was alright. To be clear, I think the more Netgate gets criticized and called out the better. But I had no hopes beyond that.
> My 'hidden' agenda is this: Because hostility takes a toll on the recipients (us) ...
Putting aside that I'm not completely on board with the hostility characterization either, you're recipients of it only in the sense that you happened to read it. I disagree with you about the degree to which Netgate deserves the criticism of course, but none of the "hostility" was addressed to you or anyone else in this thread.
It shouldn't be taxing. It's pick-me-up to anyone who's read one too many overly positive comments about the pfSense Plus shenanigans.
Like you, i have used pfSense since the 1.2.3 days...which is about 2008-2009 or so. I even bought the book to support the devs at the time (which to my knowledge have left for greener pastures). In some sites I even replaced failing hardware with a legit appliance. And even with COVID, pfsenese allowed me to quickly spin up OpenVPN appliances as standalone boxes (something i tried on OPNsense but couldnt get stable, largely due to the interface changes and my lack of familiarity with them). All of that is to say that I have been a big supporter of theirs, having submitted small bug fixes pre-netgate days and even buying/financially some of their later endeavors.
But the issues are as much
1. Starting with the 2.4 train, you can no longer really compile from source. Their build.sh relies on some closed source components not in their git repo. Specifically a small program called gnid that creates a unique ID and AT LEAST calls home to netgate to report that. They have been very cagey about what all occurs but it does happen outside of the firewalls application itself (ie: you cant block it with a state rule). Bringing this up in forums brings in ad-hoc attacks and open hostility. Gonzo is on-record saying if you cant compile its because you dont know what you are doing or something of the sort.
2. They are openly hostile to FreeBSD, forks like OPNsense (which at one point they squatted a similar domain and even tried to spread amlicious misinformation). https://opnsense.org/opnsense-com/. Theres more...entire threads of nonsense and reading. its out there if you want...But all that is to say...everyone has mud of their face when its slung around like it has been.
You may say this is childish and so comically so theres no way its true. But if you see how they conduct themselves on reddit and listservs its actually somewhat inline.
3. Finally, when gonzo or whatever his name is started back into the project and spawned netgate that was mainly to sell certified appliances as a means to support development. Initially he attacked storefronts on sites like amazon that would pre-package the Community edition onto supermicro boxes etc. And that seemed reasonable (at least to me), even though it was kosher within the terms of the Apache license.. But then with 2.5 they initially announced it would require AES-NI, which a lot of these low power boxes dont support. They backed off of that and eventually said it wouldnt be a requirement.
Ive been on 2.3 for a while now because with 2.4 they dropped x86 and went x64 only. Ive avoided opnsense because im used tot he pfsense interface and some of its more advanced tweaks. And moving to x64 is an in place rebuild and re-import. But I held largely to see how further development shakes out and frankly I'm now spending the time migrating my config over to the primary fork.
2.6 (well their move to year.month releases) will diverge from their "Open Source" code with no promises for them to stay near track. Basically its going closed source. And while they claim its up to community for further support, they also hold the keys to the PR and commits/merges....so they have the ability (and given their history) to deny commits for features/bugs that would conflict with their closed source aspirations.
From the announcment below
>In general, features that are part of FreeBSD or the other open source components that comprise pfSense will be upstreamed to those projects and made available to pfSense CE. This includes features mentioned above, like improved packet filter performance. Some features that we add to Plus will contain code that is part of these open source projects and also GUI or middleware modules that are part of pfSense Plus. In those cases, the open source code will still be contributed back and made available to CE, but work will need to happen in CE community to enable it.
Community Edition will diverge from Pfsense+ with the 2.6 release. They have also made no commitments there will be any releases after that - "it's up to the community".
They will, however, gatekeep what features the community is allowed to add. Community Edition is more or less a dead man walking at this point, they just refuse to come right out and say that.
Someone asked if they'd allow one of the REST API projects to be put into upstream and they gave some ridiculous answer about how they'd review any commit but alluded to the fact they won't actually accept it. Because what would they do if the maintainer left? Their suggestion was to fork it. Which, ironically, is exactly what OPNsense did and then Jim Thompson acted like a misbehaving 6 year old and created a website trying to bash them and didn't even have the spine to own up to it until there was a court order.
I'm not sure why ANYONE would waste any effort on adding anything to pfsense at this point when they won't actually commit to accepting features upstream that competes with PFsense+.
In my case, I don't readily find hostility toward a group that has busted tail to provide me tremendous value while I have contributed very little in return. My interactions over the years have been - perhaps not exclusively positive but overwhelmingly so.
History says one day pfSense will no longer fill my needs. Okay. I'll raise an imaginary glass move on with gratitude.
Well instead of pfSense no longer fulfilling your needs than maybe its time to beam up to the mothership. FreeBSD can do everything pfSense does without a web interface.
pfSense provided a real easy of use, at least back in the day. Given that the whole config synced over to a backup/HA failover system and updates to one could easily be confirmed synced to the other, there was a real ease of use in using pfSense (at least I thought so about a decade ago when I was using it). Spend enough time configuring HA firewalls and you start wishing you had something to take care of alerting about config differences and syncing changes automatically, and that's one of the things pfSense offered that was good.
This wasn't a case of us not knowing how to configure stuff in the OS, we moved from configuring OpenBSD firewalls with pf+pfsync, ipsec+sasync and carp to pfSense because it just made it easier to deploy and configure, given we had about ten or more of these we maintained for customers.
Even recently at a new job we were talking about upgrading or replacing some HA FreeBSD firewall pairs, and I was suggesting pfSense because it's simple to use, and just BSD underneath. Given what I've learned in this thread about the state of the project and company behind them now, I don't think I would recommend it anymore, but I still think a similar project with similar features has something to offer over vanilla BSD.
I moved over to opnsense yesterday. Just built my config in a vm. Exported. Installed the firewall and imported and setup the interfaces.
It should do all of that and seems to have a few nice features to boot. As well as a much steadier release cycle. And a security audit feature built in to tell you if the updates available will patch vulns. Which I found neat
Nice, and thanks for the heads up on your experience. I was actually just looking into comparisons of them today, because I wanted to know what the major differences were, if any. I came across this[1], which while not extremely recent, it within the last year.
Everything looks pretty good for opnsense IMO based on that. The only thing that gave me pause was the note about (unsubstantiated) reports of VLAN problems in opnsense that have supposedly been broken for a while. We make heavy use of VLANs, so that would be problematic, but it could be fixed by now or never have been the longstanding problem reported for all I know, I haven't gotten to that point because I'm not planning on anything in the immediate term that requires it.
I haven’t had any problems so far with them. (I run about 5 vlans at home).
Keep in mind I’m using intel nics (igb driver), promiscuous mode on. They seem the same as others.
The major things I’ve had to muck with.
1) NUT seems bugged. I can’t get it talking via usb as a stand-alone at all. Though I can see the APC UPS via usbconfig. Even when I just pointed it at my nut server I’m seem TTY broadcasts on the ssh session that its dropping snd reconnecting.
2) vpn configs carried over but assumptions made in PFsense had to be input in opnsense. Such as outbound nat on my full tunnel (I run manual nat). And firewall rules have to be put in, generally with the vpn cidr scope at the source address.
3) suricata is definately less....chatty than my snort config on pfsense. Again assumptions in pfsense have to be put in manually (such as specifying your external IP to $HOME in advanced). Also the new policies filters/rules doesn’t seem well documented though it’s brand new as of 21. I’m thinking et pro has less false positives than my old snort options. I’m also still in IDS mode, haven’t started dropping. Their appid implementation seems broken though.
4) php73 seems to freak out here and there. Webui can be crashy, especially big operations like hitting download for suricata rule sets.
5) traffic shaper is definately a little different. Though for me less complex and better. But I haven’t really dug in. I have seem to drops on a specific rtsp stream cross vlan. Hoping sharing rules can fix it.
Overall I like it. It’s a nice improvement despite the bugs.
Except it's not. The source that is provided doesn't actually build pfSense as shipped. Plus there are binaries that no source is provided for that "you don't need to worry about"
Wow. Netgate come off as incredibly unprofessional.
According to the article linked and the info here in that email you linked this is my conclusion:
* Netgate tried to ship flawed code that has multiple security issues.
* Jason Donenfeld, one of the lead Wireguard developers, went out of his way to work on rewriting it to be better in time for the 13.0 release of FreeBSD
* This Netgate employee is angry that they weren't able to ship their bad code and starts throwing accusations of a smear campaign.
Am I understanding what happened correctly? Because it really makes this Firewall/Router look really bad.
That was my impression too, then I went back a couple prior messages, and looked at the earlier announcement. Wihle Netgate looks to have overreacted (at least from the info we have), I can understand why they would be upset. This was in the original announcement:
The first step was assessing the current state of the code the previous
developer had dumped into the tree. It was not pretty. I imagined
strange Internet voices jeering, “this is what gives C a bad name!”
There were random sleeps added to “fix” race conditions, validation
functions that just returned true, catastrophic cryptographic
vulnerabilities, whole parts of the protocol unimplemented, kernel
panics, security bypasses, overflows, random printf statements deep in
crypto code, the most spectacular buffer overflows, and the whole litany
of awful things that go wrong when people aren’t careful when they write
C. Or, more simply, it seems typical of what happens when code ships
that wasn’t meant to. It was essentially an incomplete half-baked
implementation – nothing close to something anybody would want on a
production machine. Matt had to talk me out of just insisting they pull
the code entirely, and rework it more slowly and carefully for the next
release cycle.
I can understand being upset if that's how you're portrayed publicly.
Keep in mind, back in February of 2020 when Kip Macy first announced that Netgate had hired him to port Wireguard, Jason offered to help. First Kip declines the offer, then seems to warm slightly to it, but ultimately appears to have not actually engaged Jason.
If I'm Jason and I offer my help (for free), they don't take me up on my offer, then try to release code that would make my baby look quite ugly, I would probably also have a pretty severe reaction.
Could Jason have been slightly more professional? Absolutely. But we're all human and I can't entirely blame him, I'm sure he was frustrated that he offered to help multiple times and they both didn't take him up on the offer, and tried to release a hatchet job with his name (indirectly) attached to it.
> Could Jason have been slightly more professional? Absolutely. But we're all human and I can't entirely blame him
Oh, I don't entirely blame him. I just partially blame him for not seeing the obvious way this could devolve into a problem, even if it would (justifiably) seem unlikely to go to this level so fast. That is, he shouldn't be surprised there was a problem with what he said, although the scope of the problem is a bit more than I think most would expect.
Professionalism isn't just about making others feel good, it's about optimizing for useful outcomes, which includes covering yourself. Not taking care with your words is just like not taking care with your code. Sometimes there's a weird interaction and things go boom.
There's not a good way for me to respond to that without going off-topic. The following is assuming that wasn't a rhetorical question, if it was rhetorical I guess we may just agree to disagree:
Until he issues a public apology for his actions, I'll refer to him as Kip. Changing your name to run from the google searches is completely understandable, and I support second chances, but you need to show a bit of remorse IMO.
Wow, I was thinking the headline was underselling the story before I got to the part about him fleeing the country after his parents bailed him out of jail to the tune of half a million.
I don't really think that the 'online mob' has the right to hold someone's past actions over their head, and expect some public appeasement before it relents.
The actions of... pouring ammonia in his tenants' beds, throwing their stuff onto the street in trash bags, cutting holes in their apartments' floors while they were inside, cutting through floor joists under their apartments and physically attacking the building supervisor when he complained, fleeing the country to avoid arrest and sticking his own mother with a half-million-dollar bail forfeit as a result...
... no, no, there's no reason to hold actions like that over someone's head. It's entirely praiseworthy and I'm sure it's really easy to cooperate with such an upstanding character.
Sounds like Jason should trademark Wireguard (the name). Or build an alternative brand. That way Netgate's actions, or the actions of other wireguard implementations, will not reflect on the reputation of his project/product/technology.
He did trademark the name. I don't think Jason is going to tell the FreeBSD project that they can't use the name "wireguard" for their implementation of "wireguard" just because Netgate put out shoddy code. It's not the FreeBSD project's fault.
I think he should exactly tell the FreeBSD project to not use the mark if they cannot meet the quality requirements (especially if the quality issues were known prior to shipping). That is assuming they actually shipped the known-problematic code, though. Which they did not do.
I don't think this is an accurate description of the responsibility hierarchy. If in fact the code by Netgate or associates/ contractors/ employees of Netgate is not of professional quality, it has no place in the FreeBSD codebase ready for a stable release. Ultimately, the FreeBSD core community/ developers are together responsible for the codebase even if you cannot hold them legally accountable because of the license. They together hand out and take back (deny) the commit rights (bit, whatever). If a highly sought after component _in the kernel_ is (at least to Jason's account) not even up to the lowest security and code quality standard it has no place in the code base in preparation for a stable or probably even a beta release. Other developers (and that's where Jason is completely in the right if his account is correct) should protest the inclusion of such possibly very bad code into the codebase more or less in late preparations for a release as I understand it.
So it is first and foremost on Netgate, if Jason is right but right after that it is on the other responsible FreeBSD co-developers. I mean having so obviously bad code in any kernel of a modern operating system release would be really, really bad. There are many people and companies dependent on it that cannot really influence anything but have to endure the consequences either way. I mean, if you buy a storage appliance, a router or a firewall you trust the quality of the product to a degree and cannot really audit much even if you had the skill. You have to take the word for it and make some reasonable accomodations. No insurance is going the replace the full damage due to lost data to an attacker or a bug. Peoples lives sometimes indirectly depend on the full chain of competence and no insurance can resurrect the dead or right the good name of anybody. Remember, most of the time when you have to update anything for security reasons, somebody didn't understand the system fully or just plain messed up. The only exception is when the problem or times / requirements have changed (e.g. the computers got so fast, we have to transition to longer keys/ passwords whatever).
So yeah, if Jason's account is accurate it is bad the code landed in the codebase at all and raises questions about the quality and security of FreeBSD. I mean, it is code directly meant for a secure-as-possible VPN and something that often directly interacts with the open internet. Surely such code should experience extra scrutiny.
From the short personal interaction with Jason he came across as quite thoughtful and knowledgeable. Over the years, he and his supporters were able to convince many not so easy to convince people about the quality of Wireguard and some of its implementations. He and the supporters have shown a long term commitment and I am for these reasons inclined to trust Jason's judgement as well.
I think this is my favorite comment in the whole thread. The reasons you outline are exactly how I feel when it comes to priorities here, and how I feel his conduct was -- Despite everything maintaining friendliness while being attacked for making technical criticisms was incredibly commendable.
I duno, if true about the code I find it very difficult to empathize with Netgate
From what has been said it's not like they found and fixed a subtle and cryptic vulnerability in an otherwise reasonable implementation and then failed to disclose it properly. It's more like they turned over a rock and found a murder victim. The guy from Netgate is also coming across as very inward looking and seems to assume everyone else's motivations are also purely selfish (referring to his comment implying a "shower of contracts" they might receive for the publicity). His focus should be on how to prevent this mistake from happening in future.
I mean, Wireguard in these cases is a piece of code _in the kernel_, is meant to be directly in the path of a packet coming from an untrusted network and potentially does cryptography/ security critical stuff. Besides maybe memory management and such things, with what other pieces of code do you care more about quality and security than basically a VPN? People and companies that don't even know what Wireguard or a VPN for that matter is directly or indirectly depend on this and similar code being nearly perfect sometimes with their reputations or even lives. I mean, it is not a game that is taking too long to load or something, which is quite upsetting but I cannot imagine how that would be critical. The security of a kernel or a VPN is a very different story and we should all stop pretending this isn't the case. I hope (and am quite convinced) Jason and others want only the best for the respective kernels and the Wireguard implementation from the standpoint of quality and security.
Even if all of the above is true, it reads like an elaborate insult. And that's fine if that what the author set out to do for some reason. Pretending it wasn't after the fact isn't being honest, in my opinion.
A more professional and neutral announcement could just talk about code that needs to be refactored due to some incompleteness and vulnerabilities.
To a much greater extent than in other security protocols, implementation security is a goal of WireGuard. The protocol itself was designed to support secure kernel implementations; for instance, it's designed in such a way as to not require on-demand dynamic memory allocation.
It's part of the premise of the security model of WireGuard that it has secure kernel implementations. If you're building a kernel WireGuard implementation for a major open source OS without taking advantage of the WireGuard implementation design concepts, you're not really building WireGuard; you're building a compatible fork and calling it "WireGuard".
The "ask" here from Jason was for everyone to slow their roll, take the flawed WireGuard implementation out of the tree, and give everyone a chance to make it more resilient. Considering the amount of work Jason had to go through to get WireGuard into the Linux tree, that seems like a very reasonable request.
Instead, the WireGuard project seems to have been put into a position where they had to scramble to fix up an implementation that was being pushed into FreeBSD, as WireGuard qua WireGuard. I can imagine that being a frustrating experience. It certainly didn't generate the most political response ever, but I think you'd be reaching to call it a deliberate insult.
My read on it wasn't that it was an elaborate insult, but more that it was far more denigrating than it needed to be, if he was trying to be professional. That doesn't mean it was purposeful, sometimes people just don't really associate the statements they make with how it may be perceived.
I think it could have been communicated clearly and succinctly with something along the lines of: "The first step was assessing the current state of the code the previous developer had dumped into the tree. We noticed some quality problems, some unimplemented protocol sections and more concerning, security issues with the code. Given these issues, we considered asking they remove the code, but instead Matt convinced me that we should rework it slowly and carefully for the next release cycle."
Notably, I think omission of the following inflammatory statements would have prevented a lot of problems:
- "It was not pretty."
- "I imagined strange Internet voices jeering, “this is what gives C a bad name!”"
- "the most spectacular buffer overflows"
- "the whole litany of awful things that go wrong when people aren’t careful when they write C."
Whether those entirely subjective statements are accurate, they are not the things you say about someone else's work output when you expect a useful dialogue with them, which is exactly why they are considered unprofessional.
I'm not defending Netgate's code here, or even the vehemence of their reaction and how they went about it, but merely noting that not only can I see how it devolved into this, I would go so far as to say it's obvious that this is why that type of language is avoided by most people trying to work professionally. Jason wrote some very unkind things, and Netgate blew up about it. There's enough blame here that they can both share some.
> The "ask" here from Jason was for everyone to slow their roll, take the flawed WireGuard implementation out of the tree, and give everyone a chance to make it more resilient. Considering the amount of work Jason had to go through to get WireGuard into the Linux tree, that seems like a very reasonable request.
Err, wasn't that actually not the ask, because he thought they wouldn't do so, so instead they worked it over in a short time-frame, only for it then to be removed when this argument broke out and it came to light?
I get your point about perceptions, but there's also another aspect of why I found it important and necessary to describe just how poor the code was:
When you're talking about replacing and rewriting the implementation on the eve of release, you better have a good reason for doing so. Stuffing a rewrite of security critical code into the kernel at the last minute is a big red flag. The main question that immediately comes up in that context is, "how is it possible that having a last minute rewrite would be better than the code that was there before? You've only looked at this for a week." And that's a really good and important question.
That much code churn is not something I wanted when I set out to get started with this, but it's ultimately where things wound up. Why? For exactly the reasons I described in my email. The idea wasn't to be _insulting_, but rather to accurately and vividly describe the state of the code, as a motivating factor for the rewrite. I see how perceptions could view that instead as denigrating, but that wasn't really the motivation. And it's not as though anybody really is rushing to defend that code either; it doesn't take a lot to look at that and make up your mind that it was probably unfinished stuff, not coded with much love, that was committed prematurely.
It also had the, I think, positive effect of leading to more scrutiny of the review process. A few people have piped up and mentioned to me that their concerns during that review weren't addressed. And as a consequence of everything, all of the code, including the rewrite, is being removed from FreeBSD until it can be carefully examined and completed, which is really the best of conclusions.
You did good, Jason. Honestly after this streissand effect from them taking technical criticism personally and threatening you, I'm probably just going to avoid anything using code they might have written... that's on them. Responding to a perceived non-professionalism by talking like that to you -- from their COMPANY EMAIL at that? If I were their boss I'd definitely start making some considerations regarding the irony of this.
I can see how someone could be insulted by that one paragraph on a personal level, especially if it had been the person that wrote the code. I can't think of a nice way to say it though.
However, on a professional level, when a core maintainer of a protocol tells you your code is bad, it's time to sit down, eat some humble pie, and start taking notes. I think that's especially true if you're offering help or are willing to address online communities with positive messaging to help them save face.
I think you handled it really well. I almost feel sorry for Netgate that they lashed out at someone so good at managing the technical (code), social (community), and political (bullshit) sides of a software project. Almost.
May I ask, how is it possible the previous code was in the code base at all when it was of so poor quality?
Btw. thank you (and all the people that probably help you here and there) for Wireguard. I put a lot of trust into You/ Wireguard and I am hardly the only one. So I am all for quality and security even if it takes as long as it takes. It is really a huge progress we have a useable and _simple_ VPN protocol and widespread implementations of it.
Sure, I didn't really interpret it as you attempting to be insulting, more that you were accidentally insulting through your explanation of what you found.
> but rather to accurately and vividly describe the state of the code, as a motivating factor for the rewrite
Sure, but is any of that really needed beyond "there were numerous security problems we had to address"? When talking about shipping crypto, I think most involved would agree not shipping it is better than shipping something possibly exploitable.
I think the core of what I was trying to express is that words should be crafted with care when expected to be read in a public forum like this, just like any code expected to be used by many should be crafted with care. For the same reason it's useful to remove quadratic algorithms from places where the input is somewhat not entirely vetted, it's useful to take care with words to reduce the chance of misinterpretation.
That doesn't mean scour your statements for the smallest possible misinterpretation, but there's a lot of room to improve things like "I imagined strange Internet voices jeering, “this is what gives C a bad name!”" while still expressing your point constructively. The low hanging fruit is easy to pick, so you might as well pick it.
To be clear, I feel for you with regards to this situation. Nobody really expects weird accusations like you got from simple emails, and that's on Netgate, but a less extreme response that also publicly notes the soured relationship would also be a negative outcome from this in my opinion, if one of lower magnitude.
Sure, but it's easy to clinically examine any communication and refine it with the benefit of both hindsight and low cortisol levels. My read of this situation is that everyone involved was stuck in a shitty situation; it got very briefly heated, and ended up where it should have: with another dev cycle to iterate on FreeBSD WireGuard.
I agree on both counts, but I think (constructive) criticism is warranted in a mistake. To absolve Jason of all responsibility would be to possibly not provide that useful feedback of why not to do this the same way next time.
Hopefully I accurately expressed that as what I was trying to convey. I don't think Jason is close to even half the problem in this case, just the small spark that allowed it to continue and explode (continue because is started with a substandard implementation to begin with). At the same time, he's also the one easier to critique constructively because the other party is hard to relate to (I'm not one to jump to conspiracy theories about implicit efforts to defame).
If the criticism is about the code instead of the person, it should never be interpreted as an insult. Proper developers have learned to split their ideas from their ego, and as such are able to receive harsh yet justified criticism concerning their code without being offended.
Similar reaction here. My first impression was Netgate being an arse. But then when you read the announcement I kind of understand why Scott is angry. Because while the post may have been in "good faith" in an Open Development and Open Source world, it surely isn't in a professional and business world especially when the work is sponsored ( being paid ).
Jason should have informed Netgate the quality of the code is shit in private and FreeBSD dev should have told Netgate will not be shipping any of it in Rel 13.
It is then up to Netgate to decide What to do with their Rel 2.5
> it surely isn't in a professional and business world especially when the work is sponsored ( being paid ).
To play devil's advocate: Netgate isn't paying Jason, and they're taking his open source code to create a proprietary commercial project. I'd say Jason owes them exactly nothing in the way of courtesy or consideration. Could he have been more polite for the sake of being polite and community goodwill? Probably.
>and they're taking his open source code to create a proprietary commercial project.
I am not sure if that is the case. Netgate seems to have used their old crappy sponsored work for their Pfsense.
That is judging from the two pieces of information here. Jason doesn't need to be of consideration for Netgate. There could be other communication we dont know about. I can certainly understand why Scott is frustrated.
>I am not sure if that is the case. Netgate seems to have used their old crappy sponsored work for their Pfsense.
Their sponsored work was based off of the Linux and OpenBSD code that Jason and others wrote. And even if it didn't utilize that code, you literally can't write a wireguard client without building on Jason's work.
WireGuard is an open-source project, and an important one. It seems to me that if you want to push to create the authoritative WireGuard implementation for a major open source OS, the commercial norms need to take a back seat.
It is unprofessional and bad business to deliberately sell your customers insecure code, and you should not expect anyone to support you in doing so. Jason would have veen within his rights to warn Netgate's customers about security holes in pfSense - but he didn't even do that, he made a comment on his own project's mailing list, intentionally not naming the company, about what he was doing and why, which is entirely within the realm of "professional."
(And there is, of course, the question of whether Netgate is in any way "professional" by building on top of an open project and not following its norms.)
NetGate spends a lot on FreeBSD development, which is great, but they also spend a lot of time running smear campaigns against people who offend them, which is ridiculous. They even started /r/opnsense on Reddit just to post shit-talking memes, and camp on the namespace to this day.
Netgate is weirdly hostile to a lot of opensource stuff, which should be strange given what all their tech is built on top of. This has been going on for years. (see opnsense etc)
They've recently forked their open and closed source products, so a lot of people have been migrating to OPNSense. I've been using it for a couple months now and recommend it.
Same. I didn't really care for open-source-ness or for the anti-OPNsense smear campaigns as much, but when they announced they were EOLing the product I used I jumped ship to OPNsense too.
It does everything my pfSense install did and then some. Eg DNS blocking and IP blocking are built-in instead of needing a pfblockerng-style plugin.
The only thing I've found worth complaining about is the accordion sidebar UI thing makes it hard to middle-click-open new browser tabs, because it's not obvious which entries in the sidebar are actual pages and which are fake hyperlinks that just expand the accordion submenu.
It seems clear to me this is a case of passionate coders with different personalities struggling with the difficult work of human communication in a world with limited resources and time.
No one has to be the bad guy here or end up hostile to open source.
I mean, Linus has openly acknowledged that he has behaved unprofessionally in the emails he sends to people who are trying to contribute. There isn't anything secret here.
Btw. great gathering of people with a *BSD background or friends :-)
I think, we all should be way more cool headed. Email doesn't contain the tone and it is very easy to formulate something in a bad way when exhausted or upset. Some people use very strong words that considering their usual style aren't nearly as strong in reality.
In this case, the great question is, how did such a bad code (according to Jason) land in the code base at all? Don't other people look at it? Do the FreeBSD developers ignore concerns of other fellow developers until it blows up to beyond community communication circles?
It makes sense to me. As I understand the FreeBSD mailing list posts, it was commissioned commercially, to add a feature to NetGate products.
Must have: in-kernel WireGuard for NetGate products.
Nice-to-have: a general-purpose FreeBSD kernel WireGuard.
The scope crept, and a piece of code that might have been fit for some purpose (whatever limitations NetGate has for its network stack, like "no jumbo frames", etc) was recast as fit for all purposes.
I guess Colin would have some insight about the process by which a ports-grade kernel module gets put on track for release in the kernel itself.
My take on this is that absolutely no part of the development process that led up to this is Jason's problem. It is not Jason's job to understand or assist or comply with NetGate's product development process; his concern seems to have been, justifiably, exclusively the proposed FreeBSD OS feature. I get the sense that a lot of the friction here comes from pfSense-types thinking that any part of pfSense is everyone's problem.
Thank you for the reply. Yes, I understand Jason is basically only involved as "I have looked at this thing you propose/ have in FreeBSD and I don't like what I see, let me improve/ discuss further plans".
My question basically is: "How is it possible, the code of such quality was even seriously included in a branch of FreeBSD that would have been released if nobody would step in?" (That is if I understand the whole thing correctly and Jason's assessment is correct.)
I also think, if you dedicate something like 5 years of your life solely to kernel C implementations of a single protocol, it's probably a lot easier for you to spot problems than it is for an arbitrary FreeBSD developer.
Short answer: With a very small number of exceptions, we trust FreeBSD developers to exercise good judgement in obtaining review and ensuring the quality of the code they commit. The FreeBSD project is selective in whom it gives commit bits to, and we have a mentoring process which further ensures that developers understand the norms.
This system isn't foolproof, but it generally works very well. There have been discussions about moving to a "mandatory review" model, but there is concern that this might overly slow down development in the context of a volunteer-run project.
Maybe. It's not necessarily without reason -- if you make a lot of contributions and they are generally very well received, it's quite sensible to anticipate that further contributions will be equally well received and to be surprised if they're not.
This was made worse by the unfortunate timing -- the final release candidate is just 3 days away. Any other time, we would have gone slower, had more discussion, et cetera; unfortunately this turned into an emergency.
Jason's reply is an impressive display of de-escalation. The NetGate person's message has a lot of hostility and Jason really doesn't return any of it. Hope NetGate comes around to working with the WireGuard maintainers more in the future.
tl;dr: "The developer of the project whose protocol I'm reimplementing and whose trademark I'm using to sell things is an attacker with ulterior motives. Anyway, everyone should be respectful and leave their egos at their door. Everyone else, that is.'
This isn’t even the only NetGate debacle. There is also the mess that lead to OPNsense. All in all, I’ve sworn off buying NetGate hardware despite it looking great in theory.
> But perhaps this is a good moment to step back and ask how we got here,
and what WireGuard itself really is.
> Traditionally, network protocols are specified in a document of protocol
behaviors. Then different organizations implement that specification.
Then everybody interoperates and all goes well. In practice, it often
doesn’t go well (see IPsec woes), but this at least has been the
traditional way of doing this on the Internet, and in some ways it
works.
> But that is not the approach taken by the WireGuard project. In
contrast, WireGuard is both a protocol and a set of implementations,
implemented with a particular set of security and safety techniques.
That’s a radical departure from the traditional model, and one surely to
raise some grumbles amongst graybeards. But I believe this is a
necessary and beneficial quality for having the types of high assurance
software that is needed for core Internet security infrastructure. When
you use WireGuard, you’re not just using some protocol that is capable
of producing packets that are legible by others. You’re also using an
implementation that’s been designed to avoid security pitfalls, and that
provides interfaces for using it that mitigate footguns. In that way,
the WireGuard project is more expansive than a mere protocol project or
a mere software project or a mere cryptography project or a mere
specification project or a mere interface project. It combines all of
those things into a single unified approach. (For this same reason, the
original WireGuard paper [2] has been difficult for folks to categorize.
Is this a systems paper? A networking paper? A crypto paper?)
> Because of that, I think this was an understandable predicament. After
all, why shouldn’t a company be able to task a developer with writing
some ring-0 WireGuard code in C? And why does it matter to me whether
the code is garbage if it can at least produce protocol packets? The
reason is that the WireGuard project’s mission is wider than that. We
deeply care about code quality and implementation particulars.
> While we now have the FreeBSD code in a maintainable state, there are
other projects too that could use some attention from us.
I get that WireGuard is his baby, but sweet jesus...
Yes. The code you're running is described as having "random sleeps added to “fix” race conditions, validation functions that just returned true, catastrophic cryptographic vulnerabilities, whole parts of the protocol unimplemented, kernel panics, security bypasses, overflows, random printf statements deep in crypto code, the most spectacular buffer overflows"
That's just horrifying. It shows someone who knows next to nothing about multithreaded code and is kludging their way through. Not someone you want within a hundred feet of anything other than maybe front-end web, and even there they're going to be the kind of person who blocks the node.js event loop (because async coding is like the junior cousin of multithreading).
It's a userland implementation. This is for the in-kernel implementation. It should be faster. Also, there are some comments that the userland version is rather hacky and probably should be transitioned away from once you can.
"pfSense® Plus software version 21.02 and pfSense Community Edition (CE) software version 2.5.0 include a major OS version upgrade, a kernel WireGuard implementation..."
The userland version is also from the original author of WireGuard and not that bad actually.
I'm currently running it in an OPNsense box to serve our internet needs. I have a connection that without VPN can push through about 400-800 Mbps, and when I put the VPN on for all traffic, I can still push 400-800 Mbps through my connection.
The in-kernel version can do the same with less CPU usage, and can probably drive multi-gigabit connections without any trouble.
"Unfortunately, the public discussion has also veered into vague claims and slanderous attacks. This is where the lack of transparency, the lack of respect, and the inflation of ego is damaging and unproductive. We had hoped for a better collaboration than this, and it makes me doubt the motives of the attackers. And yes, I make deliberate use of the word “attacker” here, because that’s what this is, an attack on Netgate and on the FreeBSD and pfSense communities. Beware of anyone who says that they have all the answers. I also worry about the integrity of those who make vague statements and blanket, over-the-top accusations."
Wow. I'm a complete outsider to this, not using FreeBSD or pfSense or Wireguard - but this blog post makes Netgate seem incredibly unprofessional. Especially to anyone who actually read the mailing list exchanges.
I think this is all pretty much over now, right? FreeBSD is pulling back from a kernel WireGuard I think everyone agrees wasn't ready for prime time in mainline FreeBSD, and everyone's working getting it ready for a future release.
I don't really understand what pfSense had to gain from a post like this, but, it's their blog.
OPNsense is criminally underrated. My main routers for my office are virtualized OPNsense VM's in high availability with CARP, DHCP, DNS, VPN endpoints, inter-vlan routing, gateway policies, outbound nat... I could go on. It all works extremely well I can't fathom why people still choose pfSense with all of the community shenanigans and closed source versions.
My only gripe with it over 3 years has been the documentation on their API's for programatically updating firewall rules/aliases could use some more examples, or just mention "use browser's network requests developer mode to see what calls you need to make".
I did LOTS of research on what firewall/router distro to install to my new router a few months ago. See my comment history for considering different options.
I have to say choosing OPNsense has been a great choice. All the things you said I can agree on, but I have to add one more thing:
That quick search bar on the top-right corner where you can quickly type where you want to go. That thing is just super nice when jumping through places in the router.
Now if I'd need to build a new router, I'd like to try my luck with NixOS. Would be great if I could just build a new router from a reproducible configuration.
Same here but I've concluded that there is nothing better than a simple install of pure OpenBSD or FreeBSD and setting the rules on /etc/pf.conf. Its safer, faster, lighter and I could argue that is also easier to admin with just SSH and no web code in between.
For example, in the latest version of OpenBSD which has a Wireguard kernel implementation, the management tool has been basically included in the ifconfig command.
ifconfig wg0 create wgport 5180 wgkey ...
And then you are set. For persistence you create a /etc/hostname.wg0 file containing the commands to bring the interface up.
I run openbsd virtualized on proxmox and it’s fantastic and not that difficult to set up (I’m a casual tinkerer at best). I’ve got a gigabit connection and can saturate that without any significant stress on the single core that it runs on.
When I came into FW distros, my practical choices were MonoWall, SmoothWall and pfSense. IPfire wasn't even on the scene yet. pfSense won me early. I figure there are a lot of similar stories of pfSense being there for us when not much else was.
OpenWrt is more of a replacement for the market routers. It's a nice Linux-based router distro with a good/great ui in LuCI. The downside of this is that upgrading OpenWRT is a bit similar than upgrading a closed-source OS of the consumer routers: you flash it and you must reinstall all packages after the upgrade. This means an upgrade between major versions is maybe a bit too much of work.
OPNsense/pfSense have similar upgrade strategies as FreeBSD has: you upgrade the core os to the latest version, then all ports. This is usually a really simple and kind of boring system, which is something you really value in a computer that manages your whole house's internet traffic...
Is that based in any way on the relative capabilities and security track records of OpenWRT and pfSense, or is a highly security-conscious userbase merely what's left for pfSense after eliminating anyone who wants good WiFi support and the ability to run on cheap commodity consumer appliances with tight memory and storage limits?
For obvious reasons (targeting embedded devices with limited flash) it's not the default, but for devices which can support it, HTTPS is easily enabled.
opkg install wget ca-bundle
sed -i "s/http/https/" /etc/opkg/distfeeds.conf
Done. And now packages have cryptographic signatures verified and are downloaded over HTTPS.
I didn't downvote but it could be you got some downvotes because calomel has a bad reputation among BSD people. They have put bad and dangerous advice in their tuning and performance posts. People who follow this advice and shoot themselves in the foot sometimes come to the mailing lists looking for help, and it turns out their problems were caused by copy pasting from an unofficial source instead of reading and understanding the documentation.
OSX changed its name 5 years ago to macOS.
For what it's worth, I've also had great success with pfSense. Ran it for years at our company. Recently we've migrated to Mikrotik, but to be honest I fail to see any major advantage. It's perhaps easier to train people in learning to use Mikrotik.
BSD adheres the POLA principle and is serving many PB of data in production at work. Rock solid and no sudden changes. The manual pages are to me of higher quality when compared to Linux.
