A DNS realization
One thing I did today was set up DNS for a hostname that we may need to re-point elsewhere very rapidly. This caused me to realize something important:
Setting low TTLs doesn't mean squat if you can't cause secondaries to reload on command.
Low TTLs mean that people will re-query A records frequently, but that doesn't help me change where the traffic is going if my secondaries haven't updated to my new set of A records. Unfortunately, none of the secondaries for our domains are under my control, and at least one of them doesn't act on DNS notifications.
The way around this problem is to make a subzone without secondary nameservers. Fortunately I could pick a more or less arbitrary hostname. (Even if you can't pick an arbitrary hostname I suppose you can usually make the fixed name a CNAME into a new subzone.)
I'm glad that I realized the impending problem while I was sitting around drumming my fingers as I waited for the secondaries to pick up the just-added hostname. Running into it during a frantic attempt to shuffle traffic destinations would have been un-fun.