Ignoring for a moment the fact that they hard coded an expiration date:
This serves as a good reminder of why it’s a bad idea to set expiration dates at the end of a year, where lots of people are on leave for the holidays (this notice came out only 2 days ago). Instead try sometime less complicated, like mid March or Sept.
It's a good reminder nonetheless. I was on a team responsible for a web server farm and we noticed after a few years that the expiration date kept creeping closer to January since we would regularly kick off the process a few weeks before it actually expired, and decided one year to renew twice just to reset the clock away from January.
Why can't you be grateful for the gift of canceled vacations and overtime pay? Cisco cares - making sure you're home (in your data center) for the holidays.
Add to that making production changes two days before holidays. Sitting here waiting for another team to debug what they broke and how to fix it so my app will work. Holidays should have started an hour ago.
x.509 certs all have a validity period. Of course they could have set them to 2099, but as far as I know you cannot change the validity without reissuing the cert.
It's called notAfter and since it's part of the signed document you cannot change it without invalidating the signature. In a custom system the issuer could (but probably shouldn't) sign a document which is identical save for the notAfter date, but in the Web PKI this would be a serious miss-issuance and have consequences.
By convention the notAfter field is filled out in Zulu timezone (ie UTC) and so these will expire simultaneously around the world.
The problem here isn’t that the certificates themselves have an expiration date, it’s that the code that generates them has a hard-coded date. Even if you try to generate a new certificate after 2020-01-01, the expiration date will still be 2020-01-01 on the new certificate.
What I find bizarre is software that accepts a self-signed certificate only to then reject the cert because it has expired. What security problem are they trying to solve there?
The reason to expiration dates on certificates is so that a compromised private key has a limited lifetime of applicability, under an assumption that you will be actively rotating keys and issuing new certificates that expire at later points in time. As far as I would think this consideration has nothing at all to do with whether the certificate is self-signed.
The model in most software that allows self-signed certs is that you eiter disable checking all together, or that the user accepts a specific cert.
If you disable checking all together, then an attacker doesn't need to compromise a private key. Any impersonation attack will just work.
If you require the user to accept a specific cert (and store that information) then you can just as well let the user accept an expired cert.
The current situation just creates a lot of hassle for people without any security benefit.
Key rotation is completely separate from the lifetime of a cert. You can renew certs while keeping keys constant forever. In practice, if your key gets compromised and you have to wait for the cert to expire, then you have a very big problem. Even with letsencrypt it is likely more than a month before the cert expires.
You can build good PKI using your own private CA just as well as using any publicly trusted root. Esp. in the realm of on-prem infra, that's not uncommon.
I think what Cisco is trying to solve is to give users the possibility to connect to the device over a secure connection in case a user has no access to a signed certificate by an authority.
So the user must still connect the self-signed certificate when connecting to the device (over ssh for example).
So in this case the user generates the certificate and also accepts it.
By the way, most ssh clients will tell you if a certificate changed. So the moment your self-signed certificate is compromised you will know.
> most ssh clients will tell you if a certificate changed
Most implementations of ssh in the wild don't use CA (or self) signed certificates. I dont know if Cisco even supports the use of ssh with certificates.
> Self-signed X.509 PKI certificates (SSC) that were generated on devices that run affected Cisco IOS® or Cisco IOS XE software releases expire on 2020-01-01 00:00:00 UTC. New self-signed certificates cannot be created on affected devices after 2020-01-01 00:00:00 UTC. Any service that relies on these self-signed certificates to establish or terminate a secure connection might not work after the certificate expires.
Certificate management is such a huge problem for big organizations. Does anyone have guidelines on how to do it properly? You really need a way to track who is using what and how they automatically get renewed a long time before expiry.
My view now is that certificates should be renewed very often - like every month, if you see something within 6 months of expiry there is a problem.
> You really need a way to track who is using what and how they automatically get renewed a long time before expiry.
If you have any sort of monitoring system in place, it should also be able to check for certificate expiration (or "validity" in general). Even the old antiquated nagios has had this ability since the dawn of time.
As a sort of last resort and/or where no other solution is in place, you can do what I did on a customer's network a few years ago. Fortunately, there wasn't much "security" in place (internally, at least).
nmap can easily 1) detect/identify TLS and 2) produce "grepable" output.
I simply gathered a list of all IP subnets that were in use and scanned all of it. Afterwards, it was fairly simple to go through the results and figure out which hosts were running services using TLS on which ports.
There are existing tools that can connect to services using TLS and gather the information from the presented certificate. If you can't find one that does what you need, that's no big deal, it really doesn't take much to write your own tool just for this purpose.
(In fact, something like this -- running on a host that has access to connect to all of your TLS services -- might be nice to have even if you have a "proper" solution in place. If (when?) something falls through the cracks, you'd have this as a sort of "fallback" or "backup" to alert you to anything your proper solution might have missed.)
I'd like to know as well. At my work we had to partner with a major bank and they demanded a fully signed (sigh) client certificate for the integration. The way their API works means that we have to do a planned hard-cutover before the certificate expires. There's a 60% chance that we'll have trouble reaching out to the right people when the time comes.
This is a common problem in healthcare insurance. API connections will stop working for about 3 weeks every year as each side performs a complicated dance of noticing the expiration and updating the systems to accept a new public certificate.
A short period of renewal would mitigate the problem where every employee on the notification list has become a former employee / invalid email by the time the actual expiry occurs.
That kind of turnover seems crazy. You should also be using an email distribution list as the "notification target" which is kept up to date when someone joins or leaves the team...
We take a belt-and-suspenders approach, mainly because I have a huge fear of the embarrassment of letting a cert (or domain name) expire.
We built a DB to track them; nothing special, it sends email reminders. Been meaning to just publish calendar reminders, but haven't gotten to it.
We also check expiry dates on the live certificates via one of our monitoring systems (Zabbix, but this isn't a built-in check) that will start yelling, I think, 2 weeks before expiry.
You can use something like Vault with the PKI backend and an audit engine set up. You will get a full trace of all certs delivered and an API endpoint to update them.
You should know your assets. If you know your assets, it's pretty trivial to write a script or program that checks when their certificate expires, sticks it in a database, and alerts you when it's getting close to time to renew. Of course, if you don't know your assets, you hopefully at least know your IP space and can scan that.
Fully automated issuance and rotation on the order of hours. Use stuff like SPIFFE [0]. Won't help you much with 3rd-party proprietary hardware but you can at least avoid dealing with long-lived certificate management for your own software.
Doubtful. Cisco software code quality isn't the best, even in their specialist fields (see the number of BGP related bugs that are patched every year, still). This looks like just another stupid bug that has already been fixed in a future release.
Yup. They have lots of smart people, but something has been badly wrong on the code quality front for decades. I remember not long after the turn of the century doing inter-continental live debug sessions with their team because the multicast IPv6 acceleration (MLD snooping, like IGMP snooping but for IPv6) they'd shipped was completely broken in our Catalyst 3750s. As usual the authorised workaround was to disable acceleration, destroying the value added by Cisco's incredibly expensive product so I was on Transatlantic conference calls after hours trying to get them to reproduce the issue so they could actually ship us a fix.
My theory is that the business is more focused on getting contracts and sales, and getting in some useless magic quadrant than they are on actually providing a quality product.
Care to explain how you think this issue is even remotely related to the grey market for Cisco products?
I'm having trouble trying to come up with any link between the two.
---
In my opinion, with the exception of perhaps very small networks, no one who's "doing it right" should really be affected by this issue.
I can kinda maybe understand folks using self-signed certificates for VPNs but, really, if you're doing much at all with certificates you should either be running your own internal PKI and/or getting certificates by somewhere else.
My previous employer had literally hundreds of devices using their own self-signed certificates (for device management) on them. Many of them were the "default" certificates that shipped on the devices, so it was the same certificate shared across hundreds of devices. As someone who gives a shit about security, that drove me absolutely nuts but no one else gave a damn. (Even worse, since nearly all of these devices used the same credentials, you'd only need to MITM one of them in order to "pwn" damn near everything on the entire network!)
The certificates on the Cisco WLC only lasted 10 years, you have to upgrade the WLC and all the access points to a new version that includes a command to turn off the certificate check. If you want to upgrade an AP after that date you need to set back the clock first, so much fun
They seem to be highlighting SIP/VOIP as the main issue. And downplaying ssh access into the router based on an assumption that most don't use x509 certs for ssh. Does that sound right?
Also, you aren't affected if you actually have a CA. This only blows up if you use the Cisco device's own self-signed certs which it always mints (in affected versions of IOS) with notAfter 2020-01-01.
For stuff like web tools that might happen by least effort, but for SSH it's a bunch of extra effort, 90% of the effort of doing it properly, yet with almost none of the security benefits. Anybody who cares will use their own CA and be unaffected, anybody who doesn't care never enabled certs for SSH on the Cisco boxes.
I did data recovery on an external drive in 2011. It was pretty bad but I only needed a dozen files or so, rest was backed up. One of the files restored had a borked creation time in May 2020. For some reason I never fixed that, so whenever I accessed that folder and sort by date, that file would be at the top. This is will stop happening soon. I grew so fond of this quirk that I'm almost sad.
It's probably going to take me about 2 months after the new year stop writing/entering "2019" on forms. And checks. And anything else that calls for a date.
Pretty much the same thing happens to me every year.
This serves as a good reminder of why it’s a bad idea to set expiration dates at the end of a year, where lots of people are on leave for the holidays (this notice came out only 2 days ago). Instead try sometime less complicated, like mid March or Sept.