2022-01-14
Link: Histograms in Grafana (a howto)
Histogram evolution: visualize how a distribution of values changes over time (via) has the article URL slug of 'grafana histogram howto', and the slug is quite accurate. It's a step by step walkthrough of how to do this for a native Prometheus counter histogram metric, which most of them are. It includes copious screenshots, which is especially useful since you have to do all of this through Grafana's GUI and describing GUI actions in text is not necessarily ideal. I've slogged through heatmaps and histograms in Prometheus and Grafana, and this article still taught me something quite useful that I hadn't realized (the 'exclude zeros' setting; I agree with the author that this should be the Grafana default).
PS: Contrary to what the article suggests, heatmap legends aren't always useful, at least in current versions of Grafana. I tried putting a legend on some disk IO latency heatmaps that have very small latencies and the result was not all that readable or clear.
Understanding what a DKIM (spam) replay attack is
I recently read A breakdown of a DKIM replay attack (via), which introduced me to the idea of a DKIM (spam) replay attack. In a DKIM spam replay attack, an attacker arranges to somehow send one or more messages with spam content through your system, and then saves the full message, complete with your DKIM signature. Once they have this single copy, they can use other SMTP servers to (re)send it to all sorts of recipients, since in SMTP and in mailers in general, the recipients come from (unsigned) envelope information, not the (signed and thus unchangeable) message.
As Protonmail notes, the damage is made worse if the attacker can
somehow persuade you to create a DKIM signature that doesn't cover
headers like Subject:, From:, or To:, for example by omitting
them from the initial message they send. If the DKIM signature
doesn't cover these headers for whatever reason, the attacker can
add them after the fact and the message will still pass DKIM
validation, and mail clients (and mail systems) will probably not
flag that the message Subject and other things being shown to people
is not actually signed. The attacker can also add an additional
Subject: header (or other headers) to see if the recipient's overall
mail system validates the DKIM signature with one but shows the
other.
DKIM signatures can be made over missing headers, which can be used
to 'seal' certain headers so that additional versions of them can't
be added. When I experimented with our Exim
setup, which uses default Exim DKIM parameters,
it did sign missing Subject: and To: headers, effectively sealing
them, but it doesn't currently seal any headers against additions.
(Exim takes its default header list to sign from RFC 4871. That's been obsoleted
by RFC 6376, but
our Ubuntu 18.04 version of Exim is definitely using the RFC 4871
list, not the RFC 6376 list, since it signs including headers like
Sender:, Message-ID:, and the MIME headers.)
Finding out about DKIM replay attacks has made me consider what we might do about them. Right now I can't think of very much we could do (although I can think of a certain amount of clever ideas for bigger, more complex places with more infrastructure). However, perhaps we should have a second set of DKIM keys pre-configured into our DNS and ready to go live, so that we can switch at the drop of a hat if we ever have to (well, with a simple configuration file change).
(I think that rotating your DKIM keys regularly might help to some
degree, but my assumption is that someone who manages to get your
to DKIM sign a bad message is most likely going to start their mass
sending activities almost immediately. If nothing else, the longer
they wait the more out of place the message's (signed) Date:
header will look.)