Wandering Thoughts archives

How not to set up your DNS (part 3)

Presented once again in illustration:

; sdig ns isotech.com.cy
dns2.lordosinfo.com.
dns1.lordosinfo.com.
; sdig ns lordosinfo.com.
dns1.edynet.com.cy.
dns1.lordosinfo.com.
; sdig ns edynet.com.cy.
dns2.netconnect.com.cy.
dns1.netconnect.com.cy.

At the moment:

• dns2.netconnect.com.cy is not responding,
• dns1.edynet.com.cy doesn't exist according to dns1.netconnect.com.cy,
• dns2.lordosinfo.com is a CNAME to outopos.lordosinfo.com,
• and outopos.lordosinfo.com is a CNAME to itself (that's good for bonus points).

In the face of all of this, our DNS server is currently laughing hysterically at the very idea of looking up A and MX records for isotech.com.cy. (In fact it bitches about 'protocol error', possibly due to the nice CNAME loop.)

How not to set up your DNS (part 2)

Today's examples of MX entries are all drawn from domains presented to our mailer in MAIL FROM's over the past 28 days or so. 'sdig' is a script that does a 'dig +short'. Presented in illustrated form:

; sdig mx oakhavenresort.com.
10 mail.
10 mail.oakhavenresort.com.

It's interesting that they get this simultaneously wrong and right. Is the plain 'mail.' supposed to be in some other domain?

; sdig mx ieg.com.br.
10 200.226.132.20.

Contrary to semi-popular belief, IP addresses are not valid as MX targets and will work on only a few systems.

; sdig mx km.ru
10 mx1.mail.km.ru.
100 217.174.96.26.

Well, they have their bases covered in the 'hostnames versus IP addresses' debate.

; sdig mx worldmexico.com.
10 mail.worldmexico.com.
; sdig a mail.worldmexico.com.
192.168.1.111

That IP address is in RFC 1918 address space, so no one outside of worldmexico.com itself will be delivering email to them any time soon.

; sdig mx everymail.net
0 smtp.everymail.net.
10 smtp-c01.everymail.net.
20 smtp-c02.everymail.net.
; sdig a smtp.everymail.net.
212.69.179.11
; sdig a smtp-c01.everymail.net.
10.0.3.66
; sdig a smtp-c02.everymail.net.
10.0.3.67

I consider this the grand prize winner.

Should smtp.everymail.net ever not respond, very odd things start happening. If they are lucky, people simply cannot connect to their backup MXes; however, if the sender is using RFC 1918 10.*.*.* IP addresses internally, email to everymail.net may fly off to some internal machine, possibly to drop into someone's mailbox or bounce explosively.

The good news is that this sort of thing happens only very rarely; 58 domains out of 4,399. (Of course, a certain amount of the other ones simply don't exist.)

Sidebar: People who don't want to get mail

; sdig mx viewdocs.com.
0 dev.null.
; sdig mx headbone.com.
10 127.0.0.1.

Someday our mailer will reject MAIL FROM: domains that so clearly don't want to get email. More subtle is:

; sdig mx uhaultrailer.com
10 nullmx.uhaultrailer.com.
; sdig a nullmx.uhaultrailer.com.
127.0.0.1

Department of I'm not sure:

; sdig mx mailbox.co.yu
10 mail.mailbox.co.yu.
; sdig a mail.mailbox.co.yu.
127.0.0.3

All of 127/8 is the looback address, but most people use 127.0.0.1. (They also have www.mailbox.co.yu pointing at 127.0.0.2. Perhaps they are very definitely not in business any more.)

; sdig mx wickedmail.com.
10 localhost.

That's almost like oakhavenresort.com, except more straightforward.

Puzzling and mysterious is:

; sdig mx cyberpublications.com.
1 bounce.argewebhosting.nl.
2 mx2.argewebhosting.nl.
3 mx3.argewebhosting.nl.
; sdig a bounce.argewebhosting.nl.
127.0.0.1

But the other two hostnames have valid IP addresses that even respond on the SMTP port and accept email for cyberpublications.com. One would think that argewebhosting.nl could make up its mind; does the domain get mail or not?

How not to set up your DNS (part 1)

Presented in illustrations:

; dig +short ns harvest.idv.tw.
harvest.idv.tw.
www.harvest.idv.tw.
; dig +short a harvest.idv.tw.
219.84.30.59
; dig +short a www.harvest.idv.tw.
219.84.30.59

To those setting up nameservers: when people said 'have two nameservers', they did not mean 'and feel free to give them the same IP address'.

As a bonus, harvest.idv.tw has probably doubled the amount of time many DNS servers take to give up on them when 219.84.30.59 is having a wee problem.

Making self-signed SSL certificates with OpenSSL

So that I don't have to try to remember it or look it up next time around, here is how to generate a real self-signed SSL certificate with OpenSSL. The basic but incomplete incantation is:

openssl req -x509 -nodes -days <HOWMANY> -keyout <FOO>.key -out <FOO>.crt ...

If you want a merged PEM certificate, just make the -keyout and -out arguments the same thing. The key (or the PEM certificate) must be kept private; the certificate can be world-readable. (This command creates an unencrypted private key, with no password. My opinion is that anyone using encrypted private keys for servers is a masochist.)

To the basic incantation you must add one of two sets of arguments:

• '-newkey rsa:1024'; this will prompt you for information; in order, C, ST, L, O, OU, CN (usually a host name), and the email address. It will eat standard input.
• '-new -config <CONFIGFILE>'; this will take all information from the config file.

(On some systems you can also use '-newkey rsa:1024 -subj <INFO>', where <INFO> lists the CN et al information in a compact form, but this barfs on at least some of my machines. See, for example, here. If you need to specify an email address, its attribute name is 'emailAddress'.)

A configuration file looks like this:

RANDFILE = $ENV::HOME/.rnd

[ req ]
default_bits	= 1024
default_keyfile = privkey.epm
distinguished_name = req_distinguished_name
prompt		= no

[ req_distinguished_name ]
countryName  = CA
stateOrProvinceName = Ontario
localityName = Toronto
organizationName = University of Toronto
organizationalUnitName = CNS
commonName   = utcc.utoronto.ca
emailAddress = spamtrap1@utcc.utoronto.ca 

Naturally I don't recommend that you copy this example literally, unless you really want to generate a self-signed key for our server.

Having created a PEM file (or just having one lying around), you can examine it with 'openssl x509 -inform PEM -text -noout -in <FOO>.pem'. Similar arguments can likely be used for keys and for certificates.

Sidebar: Other references

Here is a discussion about a potential problem with self-signed certificates if you have to renew them. Locally we deal with this problem by generating self-signed certificates with very long expiry times; my current choice is 9999 days.

A good Google search for this seems to be [openssl self-signed certificate]. If you throw in 'making', the top results turn into being mostly about how to make your own Certificate Authority cert and then sign things with that. This is somewhat different from a self-signed certificate, plus is more work.

A little sysadmin twitch

Every system administrator has little twitches, behaviors that strike outside observers as somewhere between odd and superstitious. Sometimes these are just habits, but sometimes they have interesting stories behind them.

One of my twitches is that when I use su, I always type '/bin/su' (or on some irritating systems, '/usr/bin/su'), not plain 'su'. (By now it's a reflex.)

This habit originated in a bit of security advice. The story goes that if you didn't use absolute paths, an intruder who compromised your account could alter your $PATH so that plain 'su' ran a trojan version of su that logged the password somewhere, told you 'bad password', and then cleaned itself away to make future su's work normally.

In my current environment this is mostly superstition, because most of the time I get root access by starting a 'root xterm' (more or less 'xterm -e /bin/su -fg OrangeRed'; the red foreground makes such xterms stand out, avoiding accidents). An attacker who compromised my account could just zap the fvwm2 menu entry for this instead of changing my $PATH.

Still, it's my little twitch. Life wouldn't be quite the same without it.

How not to do DNS for internal domains

Here's a brief recipe for how not to do DNS for your internal domains, as illustrated by eBay:

1. Allow your internal subdomains leak into your externally visible nameservers, so that when outside people query for 'sjc.ebay.com' they get back NS records instead of 'no such domain'.
2. Use RFC 1918 private IP addresses in 10.*.*.* for your internal network, including the DNS servers for your internal subdomains. Such as sjc.ebay.com.
3. Every so often, send out email with the envelope origin address of 'cmuser@hoho.sjc.ebay.com'.
4. Watch the comedy that ensues as people's mailers try to verify the MAIL FROM by querying the nameservers for sjc.ebay.com to see if hojo.sjc.ebay.com has an MX or an A record. You know, the internal nameservers with unreachable private 10.*.*.* IP addresses.

For extra comedy, consider what happens if eBay is trying to send email to an organization that is also using 10.*.*.* IP address space internally.

Since failure to reach nameservers usually causes a temporary failure during SMTP instead of a hard failure, this is really the gift that keeps on giving. (Which means that eBay pays a price for this too, since they get to sit on all of the stalled mail until it times out in four days or so.)

(This happened some time ago, so I don't know if eBay is still sending out email with those internal addresses. The domains are certainly still leaking out, nameservers in 10.*.*.* and all.)

How doing relative name DNS lookups can shoot you in the foot

DNS-based host name lookups can be what I'll call 'relative', which look for the host name inside some domain, or 'absolute', which assumes that the host name is fully qualified and starts right from the root DNS. (For clarity, absolute names are often written with a trailing '.'; this can often be used to make resolver libraries treat them as absolute.)

Once upon a time, we had an interesting mail explosion. The campus wide mail servers started sending our server bouce mail addressed to various users at 'mail.com.cn'; our server accepted it (we're willing to relay mail for on-campus people), and it promptly sat around doing very odd things. In addition to the problems, this struck us as very odd; the campus wide mail servers do not normally smarthost outgoing mail through us.

What had happened was a DNS problem combined with relative name lookups:

• various spammers sent mail to nonexistent user names on the campus-wide mail system, using various forged mail.com.cn usernames as the origin address. Bounces ensued, and the mail servers tried to route them back to mail.com.cn.
• mail.com.cn's MX was just 'www.' (they probably intended 'www.mail.com.cn.').
• the campus-wide mail servers are utcc.utoronto.ca machines.
• 'www.utcc.utoronto.ca' is one of our server's aliases.

So the absolute 'www' of the MX wound up being looked up as a relative hostname in the mail server's domain, resulting in our server. Dutifully the mailer called us up and passed us the hot potato, whereupon very odd things happened because to our mailer it looked sort of like we ought to be handling mail for this domain, except it wasn't on our list of local domains.

(You might question the sanity of mailers trying relative name lookups in general. However, users usually like being able to write addresses as 'spamtrap1@utcc' instead of 'spamtrap1@utcc.utoronto.ca'.)

Fun with DNS: a semi-obscure problem

I just helped a friend troubleshoot a DNS problem he was having on one of his home Linux machines. The specific problem was that Firefox was taking a long time to start connecting to web sites; he believed it was DNS problems.

I ran down some standard guesses:

• maybe one or more nameservers in /etc/resolv.conf weren't responding, and Firefox was having to wait for lookups against each to time out. But straight lookups with dig weren't delayed, so it wasn't that.
• he wasn't using a proxy, so it wasn't the proxy server's machine having DNS problems not shared by his workstation. (Don't laugh; this can happen, and when it does it can be head-scratching territory. It's even more fun if you're using transparent proxying.)

When I thought about what could be different between dig and Firefox's lookups, the penny dropped: because dig is a tool for direct DNS queries, it ignores search directives in /etc/resolv.conf.

Lo and behold, there was a search directive that pointed to an obsolete ISP subdomain, roughly 'search gone.b.isp.com'. While the nameservers for 'isp.com' were fine, about half the listed DNS servers for 'b.isp.com' didn't respond. So Firefox's lookups for 'foo.com' started by looking up 'foo.com.gone.b.isp.com', and at least some of the time this would try to query the non-responding DNS servers for b.isp.com. Hence, timeouts.

Today's moral: if you put someone's domain in a search directive, you're at the mercy of their DNS competence for all your lookups (not just ones for their domains). And, unfortunately, people screw up DNS records and nameservers all the time.

(My friend didn't actually need any search directives, so we solved the problem by just taking it out.)

Fun with upgrading our backup server

We've been having odd problems with backups for more than a week now. Problems with backups are often frustratingly hard to debug, since generally I only get one run a day and it really should work.

Our Amanda backup server still ran Red Hat 7.3 for various reasons (including lack of disk space, which is a long story). I had been planning to upgrade it to Fedora Core 4 just before the backup problems hit, then they pushed that back, then we worked out that they'd started right after we rebooted the server to put the first of two larger system disks in so suddenly it was upgrade time again.

Apparently the server didn't like this idea. The problems we encountered included:

• a small mistake in swapping in a larger system disk.
• the Fedora Core 4 install kernel paniced on startup when I tried a network based upgrade.
• the CD-ROM drive wasn't working, so I couldn't try a CD-ROM based upgrade. We wound up swapping the CD-ROM drive, at which point the network upgrade kernel liked the system again. (I love hardware. I really do.)

Once this was sorted out, the RH 7.3 to FC 4 upgrade worked fine, which is pretty impressive if you think about it; we skipped five intermediate releases.

Now comes making sure that Amanda and the tape robot (and tape drive) are still talking to each other. This has been enlivened by the failure mode of our Amanda installation: if it doesn't have enough permissions to talk to the tape drive, I don't get diagnostics; instead it just sits forever whenever it needs to swap which tape is loaded in the tape robot.

(Specifically, a shell script winds up waiting for mt to report that the tape drive is online after having finished loading the new DLT tape. I have no idea where the error messages from mt wind up, but it's certainly not the screen.)

Then we ran out of unused tapes in the tape robot, so I'm still not sure if the whole mess really works. I'll probably only find out tomorrow night (or Friday morning).

(This does neatly illustrate one reason why the backup server stayed with Red Hat 7.3 for so long. And why it is still running Amanda 2.4.2p2, released in 2001.)

Another tip: Label your hard drives

We managed to waste a nice chunk of time today with a really simple mistake.

Our plan was to one of the two mirrored system drives in our Amanda backup server for a larger one, as part of the process of upgrading its operating system. What we accidentally did instead was to swap one of the extra spool space drives out. This did not work so well. (For a start, the system was a bit upset about not being able to mount the spool partition.)

You may remember that last week I said to always label (dying) hard drives you pull out of servers. Well, today I have a new lesson: always label hard drives, at least if a system has more than one. Having labels saying 'first system disk', 'sdc spool disk', and the like would have saved us some time and aggravation.

Unfortunately I don't know what the best way to do this is. Postit notes (what we used on the hard drive we pulled last week) seem a bit fragile. Clear scotch tape holding down very small pieces of paper?

(From this the clever reader will conclude that we didn't pause to label all of the hard drives in the server after we got the disks right. We were in a hurry to get it put back together and running. (This may turn out to be a serious mistake.))