2021-04-30
Discovering outside people attempting to do dynamic DNS updates to us
We run our own primary DNS servers for DNS for our zones, both forward zones for domains we support and reverse PTR zones for our public subnets. For a number of reasons, including that our network layout means we need split-horizon DNS, we have what I would call a semi-stealth DNS master server; it's not listed in our NS records, but it is listed as the master name server in our SOA records, including (of course) for reverse PTR zones. External people are not supposed to query this machine, but sometimes they do anyway and sometimes this is the symptom of a configuration problem, like the machine appearing in a NS record. So today, I decided to take a look on our perimeter firewall to see who was trying what these days. Much like other times I've done similar things (or), there were interesting things underneath this rock.
The most surprising thing to me was that most of the traffic our firewall was rejecting wasn't DNS queries, it was DNS UPDATE attempts. When I grabbed packet dumps and decoded them in Wireshark, they turned out to be attempts by various remote IPs to add reverse lookup information for various host names. More specifically, this looked like Windows dynamic DNS updates. Some of the updates were for machines in .LOCAL, but others were for host names in various domains, most of which seem to be real domains. Some of the host names were generic, like 'laptop-<jumble>', but others had a clear organized naming scheme or were even idiosyncratic, like 'WheelOfFortune'.
(There's one domain name that doesn't exist, but the name and the geolocation of the IP address it comes from strongly suggests that it's probably an internal name or a mangling of it that escaped into the outside world. The domain is 'ABGTOWNSHIP.com', and the IP address is reported as being in Abington PA.)
After looking at this for a while, I've come up with a moderately horrifying theory for what is causing this: I think that some people out there in the world have set up their internal networks to use part of the University of Toronto's class B 128.100.0.0/16 IP address space (it wouldn't be the first time). When Windows machines on these internal networks decide to do a dynamic DNS update to register themselves, they look up the SOA for the PTR of their subnet through public DNS, determine that it's our semi-stealth master (which is listed as the SOA MNAME for our PTR zones), and send it to us.
Whether or not update requests reach us at all likely depends on which parts of 128.100.0.0/16 are being used internally. If they're using the subnet that our semi-stealth master sits on, the update requests would probably not make it onto the public Internet. This matches the pattern of PTR zones that I saw in my limited monitoring, in that they never tried to update that subnet's PTR zone.
(In one case I was able to get results that supported this. A machine on the same subnet as they were trying to do a PTR update for could not reach the HTTP port on the IP trying the update, while machines on other subnets could.)
Our parts of 128.100.0.0/16 are very low in the address space (in fact mostly right at the start of it), for historical reasons that you can probably guess at. This probably makes us unusually likely to be affected by this, since people usually pick subnets from the bottom of broad IP address ranges (witness the eternal popularity of 192.168.1.0/24).
(This is likely the same sort of thing as we saw before with CBL mis-listings. But the last I'd seen of that was in 2014, so I'd hoped it was all over by now. I should have known better, but perhaps there's at least fewer people doing this.)
There's plenty of our work that's not being done from home
By now, we have been working from home for more than a year due to ongoing world and local events. I've wound up with complicated and tangled feelings about working from home in general, but some things about the whole experience are very clear. Back at the start of July of 2020, I wrote about how the work we weren't able to do from home was accumulating. Some of that work has been done in the ten months since then, but a lot of it hasn't been, and in many ways we remain subtly impaired in our work.
We've more or less sorted out ordering physical hardware since last July, with stops and starts. But we're still significantly constrained on what we can order because it still can't be delivered to work. Only some things can be sensibly sent to people's homes and then only in mild quantities. Fortunately we're not in a situation where we have to buy hardware (or perhaps that's unfortunate).
Our use of Ubuntu 20.04 remains low, although it would be higher if not for an issue with the 20.04 version of Exim. Since we upgrade by reinstalling, often on new hardware, Ubuntu version upgrades need someone in the office and we simply haven't had that time for anything except relatively urgent machines. It would be nice to upgrade our user login servers and compute servers to 20.04 so they have reasonably current stuff (it's not the latest stuff any more), but it's far more important to do things like replace our 16.04 servers before 16.04 support ends (a deadline that we only just made).
Every one of us has a backlog of projects that need us to build physical machines. They aren't critical projects but they're ones we have to do sometime, and they mostly aren't getting done. I've gone into the office several times in the past couple of months, primarily to upgrade 16.04 machines but always with the intention of getting some work done on additional servers, and I've never been able to do it. Things always come up once I'm there.
That's another thing I didn't realize ten months ago. When we were in the office all the time, there were a whole collection of small background things that we took care of casually and in passing, the kind of thing that takes five or ten minutes at most. Naturally they aren't being done in passing any more, so a certain amount of my sporadic in-office time gets spent on them instead. We always seem to get less done in the office than we planned and it takes longer than we expected.
(One aspect of that is coordinating any work that needs more than a single person to do something, especially in person. It used to be simple for us to get something racked or the network configuration of a server shuffled around; now, not so much.)
On top of all of the things that are visibly not getting done, some things are getting done slower. We can get the work done while working from home, but it's somewhat slower and more awkward. To the extent that I even consciously notice it happening to me, it can feel like being nibbled by moths.
(I'm quite fortunate to be able to work on a lot of things in a relatively low-friction way, because I can still use my office workstation's VMs. I do a lot of things that require my test machines to be on our networks with good bandwidth, which would very much not work for a VM on my home machine.)