Seems like MikroTik and Ubiquiti haven't been particularly good on documenting how their routers work, what each process does, and how to debug/verify what they are doing. I've been particularly worried about Ubiquiti, since they seem to be ignoring the MIPS based EdgeRouter line. The various consumer NAS vendors have been a nightmare security wise, and it's far from clear that the prosumer routers from Ubiquiti and MikroTik are any better.
Should it really require reverse engineering to figure out how a Linux box passes packets?
I gave up on Ubiquiti and bought a tiny $120 router ($140 with a nice metal case) that's a NanoPi R6s. Pretty impressive specifications. 8GB ram, 8 cores (4 fast Ax76 and 4 slow A55s) and no fans. Has 2x2.5GBe and 1x1GBe for networking. I've installed a port of OpenWRT called FriendlyElec and Ubuntu 22.04.1 LTS. I didn't bother cross compiling, it's plenty fast for native compiles.
I've been impressed so far. It compiles Rust about half as fast as my Quad core Xeon server from years ago, and is SEVEN times faster than a RPi 4 8GB! I have an Apple M1 mini around that manages the same compile in 13 minutes. A nice bonus is neither network or storage is USB attached, unlike the RPi.
Burned it in overnight, running all 8 cores flat out, with no problems. Haven't decided what to use, maybe shorewall or just raw nftables/iptables.
Similarly there's 4 and 6 port 2.5 Gbe boxes with various N5000 and N6000 celerons at around $200. ServerTheHome has reviewed many of them, they seem to be evolving nicely, better cases, better heat sinks, better 2.5Gbe Intel chips, etc. Run *WRT, pfsense, or whatever else floats your boat. STH even demod running the firewall under proxmox on one of these cheap 4-6 port 2.5GBe widgets. Would be nice to keep your complete config in git or the config management widget of your choice (often Puppet or Ansible).
Guess I'm just getting less trusting in my old age.
You've thrown out a lot of performance numbers here but most of us aren't compliling rust on our routers. Have you stood up iperf to see how fast it routes? The task is harder than you think if you want any features to go with that and there's a good reason most vendors (Mikrotik and Ubiquiti included) offload most of the task to silicon.
Yeah, sure, but it's frustrating. Sure casual users who won't saturate the network get fast offloaded performance.
However more aggressive users will end up enabling more complicated firewall rules, QoS, buffer bloat mitigation, or deep packet inspection and disabling the silicon based acceleration or bottlenecking elsewhere.
Thus my interest in the RK3588, that provides 4x the PCIe of the RPi4 and a fair bit more CPU.
I do wish there was a popular benchmark that used a more realistic benchmark. Smaller packets, dozens or 100s of TCP sessions, non-trivial firewall rules, etc. Doesn't seem like you learn much from iperf managing 95% of of line rate with large packets.
I suspect the majority of consumer gigabyte routers are never saturated. The technology has eclipsed typical demand. Not that there should be deceptive marketing or performance cheating, but I suspect 95%+ of people will never be limited by their network gear.
I think building it yourself works well for routers and NASes, but there's a gap for WiFi. Lot of the bells and whistles that make modern wifi perform well (beamforming, MIMO, ...) is locked behind proprietary software and your homemade access point will never work all that well.
Interestingly, this is also something MikroTik isn't very good at - their long-range wifi is nice, but their consumer wifi used to be stuck at 2x2 and basically worse than cheap Asus "wifi router" which is somewhat embarrassing.
For switches, if you want fast, for small installation it is hard to beat the Mikrotik 4xSFP+ 10G switch for $150.
Not a single wifi consumer vendor have any say on those features. it is one hundred percent done on the closed OS in the radio chip.
your expensive ubiquiti/oanda/cisco have the same mimo/beanforming performance as any opensource/clone using the same wifi radio chipset.
the consumer facing OS just flip a bit somewhere in the configuration flash. granted, knowledge of the right bit to flip mighty be missing on the opensource still, but there's no magic happening in ubiquity owned code.
uniquity is extra shaddy as they buy off the shelf components but demand custom labeling to look like it's their custom silicon. it's not. uniquity is the matress store of wifi.
> uniquity is extra shaddy as they buy off the shelf components but demand custom labeling to look like it's their custom silicon. it's not. uniquity is the matress store of wifi.
They're not - I have no idea about the hardware itself, but what makes Ubiquiti so popular is the software integration layer. Their stuff Just Works even for incredibly complex and large installations while still being easy to configure.
I've also replaced a Ubiquiti EdgeRouter with a spare PC I had around. It was a small form factor thing, I don't need WiFi, so I replaced the WiFi card with a M.2 Ethernet card[1], which gives me two network interfaces. The point is pretty much anything is powerful enough to be a router now.
(Actually my setup is more complex, as the machine is powerful enough to run a few VMs, so I pass through one of the PCI devices to a firewall VM, which means the VM host doesn't even see the outside world.)
One thing to watch out for is Intel released some bad 2.5Gbe hardware (e.g. earlier versions of the I225-V). I even have a more recent one that seems to have issues (even when running the latest Linux from kernel.org). You can find Realtek cards though, which seem to work better (yeah, who'd have thought that would be the case).
Apparently I225 went through 2-3 revisions, and the "b3" is supposedly ok. But the newer $200 PCs are now coming with the I226, which it's claimed is much better.
If that sort of thing can work at PCIe rates I don't see any reason why that M.2 card can't. It's a lower bit rate plus theoretically a more resilient encoding for long distance as opposed to PCIe.
The untwisted ribbon cable is fine for PCIe in short runs by virtue of PCIe data lines being a differential signal and if the length of cable is kept short(-ish). The max length of the cable should vary inversely with the generation, that is, max length of a flexible copper extender for gen v1 is longer than gen v3, more length introduces more noise which increases error rates. A well terminated cable should be able to use the full bandwidth of the bus. I'd be completely comfortable running x16 flat out at max rate with one of those.
I don't have any hard-and-fast sources to back the above up, but punching 'pcie extender max length' into the Gargler should find you something to suffice.
I bought one of the Edgerouters thinking it would tie into their Unifi Web controller interface - no hells bells - got a Mikrotik like device where you need Cisco IOS level of skill to configure.
Got a TPLINK Omada ER605 router now - just as powerful but way easier to configure.
The funny thing is that when I was big on ubiquiti I always went the other way edgerouter only. mainly because I did not want to get tied into the whole unifi ecosystem.
They sold two versons of their switches and routers, the black ones(good) which were stand alone and the white ones(bad) which depended on the unifi system.
But then again I am also the idiot who ran openbsd on my edgerouters.
If you're considering straight nftables, do straight nftables. The syntax isn't bad for editing/reading directly. The userspace tool has quirks but you'll get the hang of it. I run my config through a templating engine (the same ruleset goes to cloud VMs etc).
My main router is virtualized on a Ryzen 5600G that also does other tasks. I have two other wifi APs that are just low power x86 boxes that also run Kodi.
Getting back to the original topic, I've still got Mikrotik for switches. The hardware is decent, the software is janky. I've got three different flavors of config - SwOS, RouterOS, and old RouterOS that requires a different type of config because the older switch chips were never updated to the newer config commands.
Mikrotik continues to hold their software too tightly instead of making the transition to a hardware company. They've built so much routing functionality, that's just pointless on devices that are switches. Yet there is some switch functionality that isn't available on SwOS. And SwOS is web-only with a binary config backup, rather than something conducive to centralized administration.
I'd love to find some low power white box 1Gb+10Gb switches that could run straight open Linux, or even using my existing Mikrotik switches that way if someone were to forcibly open their the environment.
Don't know about the Linux distros, but one problem with the FreeBSD versions (pfSense, OPNsense) is the mostly nonexistent WiFi support because there's no driver support for PCIe cards. You end up needing to hook up a COTS Wifi AP, which has all the same problems that a COTS router would've had.
But other than that, yeah I don't see any reason to use a COTS router when I can just make one. My current router is my old desktop PC that became redundant when I built a new one, sitting as a bare motherboard on a test bench to take up very little space, running OPNsense.
How does a common wireless AP have the same issues as a common consumer NAT router? Do APs have to worry about NAT state tables? Are they exposed to the Internet? Do they have to be configured to NAT/forward special edge case devices, like Nintendo / Sony games consoles?
Complaining about the lack of wifi drivers is pretty silly if you don't understand the advantages of a simple bridging wireless AP.
Cloud-based APs? Seriously? Nobody who actually takes security seriously would use cloud-based APs. Even using cloud services to download firmware updates would be shady as heck.
You just made my point for me. If you have to download binary blobs to your hardware, like wifi interfaces, you can have issues. Would you rather have vendor-dependent devices inside your firewall / NAT router, or just connected to the same network as your firewall / NAT router?
Nobody said APs are bulletproof, silly. Only people who don't know better would suggest or even imply that having separate APs is somehow worse. You want to keep the less secure stuff over which you have less control out of the important systems.
Heh, it came with FriendlyElec, I just used it long enough to make sure the hardware was working and stable. I always reinstall Linux from something trusthworthy upon receipt.
However all the NanoPi images are pinned to the same 5.10.x kernel, that I don't think it upstreamed, which is pretty ugly.
Armbian just had a new release, supporting a TON of SBCs, even a Risc-V board, which is weird based on the name. In any case, they support another RK3588 board, so I was going to try that. I want something using a mainline linux kernel if at all possible.
The 1x1GBe port on your system is the controller built in to the SoC. It supports jumbo frames so you can do full size packets over PPPoE if your ISP can handle that.
It is a feature with warning everywhere. You'll need physical access to the device to enable docker.
I think is still a very nice feature to release even if it's no complete and fully secure yet.
> your router is as secure as anything you run in container;
> if you run container, there is no security guarantee of any kind;
> running a 3rd party container image on your router could open a security hole/attack vector/attack surface;
I think the key difference here is that it enabled root access to the host RouterOS, which is generally not something that RouterOS permits, enables, or allows. It's why the word "jailbreaking" can even be used in a RouterOS context - it's similar to jailbreaking iOS or rooting an Android phone, where the end-user is NOT meant to have root access to the device.
AFAIK, for a long they were pretty hostile and there's still this webpage which asks a $45 wire transfer in order to receive a CD with open source components used in the routerOS: https://mikrotik.com/downloadterms.html
That might have changed now, but I still can't find an official repo. You could look for older versions like this: https://github.com/robimarko/routeros-GPL or try to contact them directly (or sue if you think you have a standing).
Blame the Germans, those were the ones responsible for the FCC Radio Lockdown.
The EU imposed software radio lockdown, there was a public consultation about it where most of the answers from industry and users were negative, but they kept on pushing for it.
The same kind of consultation happened in the US, with the same kind of ignorance.
We could have cooperative use of the airwaves and true open-source, open-hardware radios, but there's too much money to be made auctioning spectrum and assuring that most of it is is 99.99% unutilized. Would be horrible if people had cheap 5-10W handhelds that use advanced codecs, or could run 5mw of power on any part of the spectrum.
No doubt about the quality, I already have some Routerboards around and will be purchasing some of their ceiling APs and a couple switches at the new house soon. All the worries come from the software being closed; I'm not painting Mikrotik evil, it's just that closed software and firmware are the ultimate place in which place spyware, especially where all software is FOSS. Routers and firewalls are the devices seeing every single piece of data on the network, thus is mandatory for them to be trustworthy, a quality the good reputation of their manufacturer can indeed suggest, but not guarantee.
Should it really require reverse engineering to figure out how a Linux box passes packets?
I gave up on Ubiquiti and bought a tiny $120 router ($140 with a nice metal case) that's a NanoPi R6s. Pretty impressive specifications. 8GB ram, 8 cores (4 fast Ax76 and 4 slow A55s) and no fans. Has 2x2.5GBe and 1x1GBe for networking. I've installed a port of OpenWRT called FriendlyElec and Ubuntu 22.04.1 LTS. I didn't bother cross compiling, it's plenty fast for native compiles.
I've been impressed so far. It compiles Rust about half as fast as my Quad core Xeon server from years ago, and is SEVEN times faster than a RPi 4 8GB! I have an Apple M1 mini around that manages the same compile in 13 minutes. A nice bonus is neither network or storage is USB attached, unlike the RPi.
Burned it in overnight, running all 8 cores flat out, with no problems. Haven't decided what to use, maybe shorewall or just raw nftables/iptables.
Similarly there's 4 and 6 port 2.5 Gbe boxes with various N5000 and N6000 celerons at around $200. ServerTheHome has reviewed many of them, they seem to be evolving nicely, better cases, better heat sinks, better 2.5Gbe Intel chips, etc. Run *WRT, pfsense, or whatever else floats your boat. STH even demod running the firewall under proxmox on one of these cheap 4-6 port 2.5GBe widgets. Would be nice to keep your complete config in git or the config management widget of your choice (often Puppet or Ansible).
Guess I'm just getting less trusting in my old age.