I was never happy with the performance of Calix CPE. We used them heavily at my last job, and indeed customers would have all sorts of trouble that we could never reproduce when we sent a tech. My favorite little hack was that I wanted live stats from the OLTs to be in our own database so that it could show up in our support portal and internal CRM and be aggregated for general network health statistics. (i.e. when someone went out to repair a fiber, they could instantly see the customers come back online, or more often... know while they were still out in the field that they didn't fix it) I wrote a program to scrape it (by ssh-ing in, thanks golang.org/x/crypto/ssh!, because their SOAP API returned no useful details), and after running for many days... it caused the OLT to stop routing packets entirely. No Internet routing, no management interface, it just flat out died unrecoverably. Anyway, they blamed my app, so I built them a static binary of the scraper that could run on Windows (they didn't have any Linux boxes) and after much back and forth they traced it down to a race condition between the two redundant processor modules in the OLT. So much whining how it was my fault, when it was their fault.
At the ISP before that we made our own CPE. The leads on that project really understood the Internet and managed to get reasonable latency, even over WiFi. But the incumbents still seem to not know about fq_codel, or how to put more than 4MB of RAM in their devices, and the users suffer as a result. This article reminded me of how mad it makes me, sorry for the rant. (I switched to a different industry where less lasers are involved.)
The CenturyLink CPE for DSL is pretty crap too. I had one that would reboot if you sent a fragmented IPv6 packet, among other problems like the web UI refusing to work if it was on long enough (thankfully EOL and they replaced it without much questions). The replacement didn't reboot, but I was seeing ping times go up to seconds, so I gave up and do PPPoE on my equipment now (I didn't want to ever run PPPoE cause it's stupid, but now I have to)
Yeah. I feel like these products were always designed to be ultra short lifespan devices, but in practice, they live forever. ONTs from 2013 are still being used and sold new, so clearly investing in fixing bugs and testing would have paid dividends. But I guess it's that case where you can't go to Amazon and buy the best ONT; Calix or whomever sells them to an ISP once, everyone has a good round of golf and dinner or whatever, and the customers are stuck being sad (and maybe the ISPs give out token credits; we gave out a lot of token credits). The ISP mostly competes by being available to a customer; if you dig up the street, then they don't really have an option to go elsewhere unless someone else digs up the street.
Hopefully some crazy person will launch an astronomically expensive fleet of satellites that solves these problems once and for all. All you need are a few engineers on the CPE team that give a damn, and you can fix the Internet for everyone in the world.
At the ISP before that we made our own CPE. The leads on that project really understood the Internet and managed to get reasonable latency, even over WiFi. But the incumbents still seem to not know about fq_codel, or how to put more than 4MB of RAM in their devices, and the users suffer as a result. This article reminded me of how mad it makes me, sorry for the rant. (I switched to a different industry where less lasers are involved.)