2020-06-29
How Prometheus Blackbox's TLS certificate metrics would have reacted to AddTrust's root expiry
The last time around I talked about what Blackbox's TLS certificate expiry metrics are checking, but it was all somewhat abstract. The recent AddTrust root expiry provides a great example to make it concrete. As a quick summary, the Blackbox exporter provides two metrics, probe_ssl_earliest_cert_expiry for the earliest expiring certificate and probe_ssl_last_chain_expiry_timestamp_seconds for the latest expiring verified chain of certificates.
If your TLS server included the expiring AddTrust root certificate as one of the chain certificates it was providing to clients, the probe_ssl_earliest_cert_expiry metric would have counted down and your alarms would have gone off, despite the fact that your server certificate itself wasn't necessarily expiring. This would have happened even if the AddTrust certificate wasn't used any more and its inclusion was just a vestige of past practices (for example if you had a 'standard certificate chain set' that everything served). In this case this would have raised a useful alarm, because the mere presence of the AddTrust certificate in your server's provided chain caused problems in some (or many) TLS libraries and clients.
(Browsers were fine, though.)
Even if your TLS server included the AddTrust certificate in its chain and your server certificate could use it for some verified chains, the probe_ssl_last_chain_expiry_timestamp_seconds would not normally have counted down. Most or perhaps all current server certificates could normally be verified through another chain that expired later, which is what matters here. If probe_ssl_last_chain_expiry_timestamp_seconds had counted down too, it would mean that your server certificate could only be verified through the AddTrust certificate for some reason.
Neither metric would have told you if the AddTrust certificate was actually being used by your server certificate through some verified chain of certificates, or if it was now completely unnecessary. Blackbox's TLS metrics don't currently provide any way of knowing that, so if you need to monitor the state of your server certificate chains you'll need another tool.
(There's a third party SSL exporter, but I don't think it does much assessment of chain health, or give you enough metrics to know if a server provided chain certificate is unnecessary.)
If you weren't serving the AddTrust root certificate and had a verified chain that didn't use it, but some clients required it to verify your server certificate, neither Blackbox metric would have warned you about this. Because you weren't serving the certificate, probe_ssl_earliest_cert_expiry would not have counted down; it includes only TLS certificates you actually serve, not all of the TLS certificates required to verify all of your currently valid certificate chains. And probe_ssl_last_chain_expiry_timestamp_seconds wouldn't have counted down because there was an additional verified chain besides the one that used the AddTrust root certificate.
(In general it's very difficult to know if some client is going to have a problem with your certificate chains, because there are many variables. Including outright programming bugs, which were part of the problem with AddTrust. If you want to be worried, read Ryan Sleevi's Path Building vs Path Verifying: Implementation Showdown.)
Adapting our Django web app to changing requirements by not doing much
We have a Django web application to handle (Unix) account requests, which is now nine years old. I've called this utility code, but I mentioned recently that over that time there have been some changes in how graduate students were handled that needed some changes in the application. Except not very much change was necessary, in some ways, and in other ways the changes are hacks. So here are some stories of those changes.
When we (I) initially wrote the web application, our model of how new graduate students got Unix accounts was straightforward. All graduate students were doing a thesis (either a Masters or a PhD) and so all of them have a supervising professor. As a long standing matter of policy, that supervisor was their account sponsor, and so approved their account request. Professors can also sponsor accounts for other people associated with them, such as postdocs.
(This model already has a little glitch; some students are co-supervised by more than one professor. Our system requires one to be picked as the account sponsor, instead of somehow recording them as co-sponsored, which has various consequences that no one has complained about so far.)
The first change that showed up was that the department developed a new graduate program, the Master of Science in Applied Computing. Graduate students in the MScAC program don't write a thesis and as a result they don't have a supervising professor. As it happened, we already had a model for solving this, because Unix accounts for administrative and technical staff are not sponsored by professors either; they have special non-professor sponsors. So we added another such special sponsor for MScAC students. This was not sufficient by itself, because the account request system sometimes emails new graduate students and the way those messages were written assumed that the student's sponsor was supervising them.
Rather than develop a general solution to this, we took the brute force solution of an '{% if ...}' condition in the relevant Django template. Because of how our data is set up, this condition both has to reach through several foreign keys and uses a fixed text match against a magic name, instead of checking any sort of flag or status marker (because no such status marker was in the original data model). Fortunately the name it matches against is not exposed to people, because the official name for the program has actually changed over time but our internal name has never been updated (partly because it was burned into the text template). This is a hack, but it works.
The second change is that while all graduate students must eventually get a specific supervisor, not all of them have one initially when they arrive. In particular, there is one research group that accepts most new graduate students collectively and then sorts out who they will be supervised later, once the graduate students know more about the group and their own interests. In the past, this had been solved artificially by assigning nominal sponsors immediately even if they weren't going to be the student's supervisor, but eventually the group got tired of this and asked us to do better. The solution here was similar to the MScAC program (and staff accounts); we invented a synthetic 'supervisor' for them, with a suitable generic name. Unlike with the MScAC program, we didn't customize the Django templates for this new situation, and unfortunately the result does look a little ugly and awkward.
(This is where a general solution would have been useful. If we were templating this from a database table or the like, we could have just added a new entry for this general research group case. Adding another Django '{% if ...}' to the template would have made it too tangled, so we didn't.)
I don't think we did anything clever in our Django application's code or its data model. A lot of the changes we were able to make were inherent in having a system that was driven by database tables and being able to add relatively arbitrary things to those tables (with some hacks involved). Where our changes start breaking down is exactly where the limitations of that start appearing, such as multiple cases in templates when we didn't design that into the database.
(Could we have added it later? Perhaps. But I've always been too nervous about database migrations to modify our original database tables, partly because I've never done one with Django. This is a silly fear and in some ways it's holding back the evolution of our web application.)
PS: You might think that properly dealing with the co-supervision situation would make the research group situation easy to deal with, by just having new graduate students 'co-sponsored' by the entire research group. It's actually not clear if this is the right answer, because the situations are somewhat different on the Unix side. When you actively work with a supervisor, you normally get added to their Unix group so you can access group-specific things (if there are any), so for co-supervisors you should really get added to the Unix groups for both supervisors. However, it's not clear if people collectively sponsored by a research group should be added to every professor's Unix group in the same way. This implies that the Django application should know the difference between the two cases so that it can signal our Unix account creation process to treat them differently.
Sidebar: Our name hack for account sponsors
When someone goes to our web page to request an account, they have to choose their sponsor from a big <select> list of them. The list is sorted on the sponsor's last name, to make it easier to find. The idea of 'first name' and 'last name' is somewhat tricky (as is their order), and automatically picking them out from a text string is even harder. So we deal with the problem the other way around. Our Django data model has a 'first name' and a 'last name' field, but what they really mean is 'optional first part of the name' and 'last part of the name (that will determine the sort order)'.
As part of this, the synthetic account sponsors generally don't have a 'first name', because we want them to sort in order based on the full description (such as 'MScAC Graduate Student', which sorts in M not G or S).
(Sorting on 'last name' is somewhat arbitrary, but part of it is that we expect people requesting accounts to be more familiar with the last name of their sponsor than the first name.)