2024-05-17
The trade-offs in not using WireGuard to talk to our cloud server
We recently set up our first cloud server in order to check the external reachability of some of our services, where the cloud server runs a Prometheus Blackbox instance and our Prometheus server talks to it to have it do checks and return the results. Originally, I was planning for there to be a WireGuard tunnel between our Prometheus server and the cloud VM, which Prometheus would use to talk to Blackbox. In the actual realized setup, there's no WireGuard and we use restrictive firewall rules to restrict potentially dangerous access to Blackbox to the Prometheus server.
I had expected to use WireGuard for a combination of access control to Blackbox and to deal with the cloud server having a potentially variable public IP. In practice, this cloud provider gives us a persistent public IP (as far as I can tell from their documentation) and required us to set up firewall rules either way (by default all inbound traffic is blocked), so not doing WireGuard meant a somewhat simpler configuration. Especially, it meant not needing to set up WireGuard on the Prometheus server.
(My plan for WireGuard and the public IP problem was to have the cloud server periodically ping the Prometheus server over WireGuard. This would automatically teach the Prometheus server's WireGuard the current public IP, while the WireGuard internal IP of the cloud server would stay constant. The cloud server's Blackbox would listen only on its internal WireGuard IP, not anything else.)
In some ways the result of relying on a firewall instead of WireGuard is more secure, in that an attacker would have to steal our IP address instead of stealing our WireGuard peer private key. In practice neither are worth worrying about, since all an attacker would get is our Blackbox configuration (and the ability to make assorted Blackbox probes from our cloud VM, which has no special permissions).
The one clear thing we lose in not using WireGuard is that the Prometheus server is now querying Blackbox using unencrypted HTTP over the open Internet. If there is some Intrusion Prevention System (IPS) in the path between us and the cloud server, it may someday decide that it is unhappy with this HTTP traffic (perhaps it trips some detection rule) and that it should block said traffic. An encrypted WireGuard tunnel would hide all of our Prometheus HTTP query traffic (and responses) from any in-path IPS.
(Of course we have alerts that would tell us that we can't talk to the cloud server's Blackbox. But it's better not to have our queries blocked at all.)
There are various ways to work around this, but they all give us a more complicated configuration on at least the cloud server so we aren't doing any of them (yet). And of course we can switch to the WireGuard approach when (if) we have this sort of problem.