Wandering Thoughts archives

Some thoughts on log rolling with date extensions

For a long time everyone renamed old logs in the same way; the most recent log got a .0 on the end, the next most recent got a .1 on the end, and so on. About the only confusion between systems was that some started from .0 and some from .1, and also whether or not your logs got gzip'd. These days, the Red Hat and Fedora derived Linuxes have switched to lograte's dateext setting, where the extension that old logs get is date based, generally in the format -YYYYMMDD. I'm not entirely sure how I feel about this so far and not just because it changes what I'm used to.

On the good side, this means that a rolled log has the same file name for as long as it exists. If I look at allmessages-20150718 today, I know that I can come back tomorrow or next week and find it with the same name; I don't have to remember that what was allmessages.3 today is allmessages.4 tomorrow (or next week). It also means that logs sort lexically in time order, which is not the case with numbered logs; .10 is lexically between .1 and .2, but is nowhere near them in time.

(The lexical order is also forward time order instead of reverse time order, which means that if you grep everything you get it in forward time order instead of things jumping around.)

On the down side, rolled logs having a date extension means that I can no longer look at the most recently rolled log just by using <name>.0 (or .1); instead I need to look at what log files there are (this is especially the case with logs that are rolled weekly). It also means that I lose the idiom of grep'ing or whatever through <name>.[0-6] to look through the last week's worth of logs; again, I need to look at the actual filenames or at least resort to something like 'grep ... $(/bin/ls -1t <name>.* | sed 7q)' (and I can do that with any log naming scheme).

I'm sure that Red Hat had its reasons to change the naming scheme around. It certainly makes a certain amount of things consistent and obvious. But on the whole I'm not sure I actually like it or if I'd rather have things be the old fashioned way that Ubuntu and others still follow.

(I don't care enough about this to change my Fedora machines or our CentOS 6 and 7 servers.)

A Linux software RAID message flood

It's early Sunday morning, which is the time when Fedora kicks off its weekly checks of MD arrays. I know this time extremely well, not because it causes visible IO load but because I actually watch my kernel messages and software RAID floods them during this check. And by that I mean that roughly once every five seconds I can look forward to a burst of messages like this:

md: delaying data-check of md51 until md53 has finished (they share one or more physical units)
md: delaying data-check of md50 until md53 has finished (they share one or more physical units)

I was initially going to write yet again about kernel messages not being sensibly ratelimited (cf), but once I started looking it appears that this is more of some sort of bug in the code. Only my home machine generates this flood of messages; my work machine, which also has software RAID arrays, generates these messages only at the start or end of a data check. The differences between the machines is that the work machine has fewer mirrored arrays (4 versus 8) and all of its arrays are on one pair of disks (at home, half are on one pair of disks and half on the other pair, because one pair is for system stuff and one pair is for my data).

Mind you, the work machine appears to have some message anomalies as well, but it doesn't emit floods of them. Particularly, take a look at this set of warnings emitted at the start of a data check:

[392229.258284] md: delaying data-check of md14 until md13 has finished (they share one or more physical units)
[392229.374681] md: delaying data-check of md13 until md15 has finished (they share one or more physical units)
[392229.374705] md: delaying data-check of md14 until md13 has finished (they share one or more physical units)
[392229.376020] md: delaying data-check of md14 until md13 has finished (they share one or more physical units)
[392229.376022] md: delaying data-check of md13 until md15 has finished (they share one or more physical units)

That's some interesting stuttering going on there, and also some recursion (md15 is the array that just started being checked).

As I was writing this entry, it took long enough that I've now discovered that this is wrong; my work machine sees this sort of thing too. When the md15 data check finished and the md13 one then started, my work machine also started repeating messages about md14 being delayed every few seconds. Fortunately md13 is small so this didn't last long. Something is clearly screwing up here.

(All of this means I should figure out how to send a bug report or two to the software RAID people.)

On the good news front, now that I've looked into this I've discovered that the number of concurrent data checks is controlled by the MAXCONCURRENT setting in /etc/sysconfig/raid-check. Setting this to '1' is not quite correct for my home machine (which could check one array on each pair of disks at the same time), but it does make the messages go away. I'll live with a somewhat more extended data check time in order to not have a message flood.

(I don't know if anyone besides Fedora and RHEL/CentOS do this periodic software RAID check, but if your system does and you see a message flood, you might want to look for a similar setting.)

Making GTK applications in alternate locations, settings edition

Suppose, not entirely hypothetically, that you build your own copy of some moderately complicated GTK application and install it a non-standard location with the equivalent of 'configure --prefix /some/where'. In theory you can then just run the application as /some/where/bin/prog and everything will be great. As I found out today, in practice you may have a subtle problem (or if you're unlucky, a not so subtle one).

GTK applications use a system for handling their settings and preferences called 'GSettings', which uses XML schema files to define all of this stuff for a particular application. By default, an application only looks for its settings schema in the system directories, even if it knows full well that its settings schema file was actually installed elsewhere. If you don't have any version of your application installed as part of the system, your own version will probably fail because it can't find its schema at all. If you do have some system version of your application installed, your own version is probably actually using the settings file from that version.

To fix this, you need to set the $XDG_DATA_DIRS environment variable so it includes the right directory for your application. If you configured with a prefix of /some/where, your schema files get put into /some/where/share/glib-2.0/schemas and $XDG_DATA_DIRS must include /some/where/share. According to the XDG Base Directory Specification, the default value for it is /usr/local/share/:/usr/share (and this agrees with what I see in testing). So you want a cover script that does this:

export XDG_DATA_DIRS=/some/where/share:/usr/local/share:/usr/share
exec /some/where/bin/prog "$@"

(This behavior is documented in the glib-compile-schemas manpage.)

There may be other things that GTK applications only look for in system areas, but if so I haven't tripped across them yet for the couple of GTK based applications I compile myself.

(Before you're tempted to throw stones at GTK for not fully supporting installing and running applications from other prefixes, note that KDE is worse for this as far as I know. I had to resort to building and installing RPMs for my locally compiled KDE application. Maybe there's a magic environment variable for it too that I just didn't stumble over.)

My early impressions of Fedora 22, especially of DNF

I recently updated first my office laptop (which runs a relatively stock Cinnamon environment) and my office workstation (which runs my custom setup) to Fedora 22, both via my usual means of a yum-based upgrade instead of the officially supported FedUp mechanism. I feel kind of ambivalent about the results of this.

On the one hand, the upgrade was smooth in both cases and everything in both of my environments basically worked from the start. This is not always the case, especially in my custom setup; I'm used to having to fiddle things around after Fedora version upgrades in order to get audio or automatic removable media mounting or whatever working again. Instead everything pretty much just went, and nothing changed in my Cinnamon environment.

On the other hand, how can I say this gently: I have not been really impressed with Fedora 22's quality control. A number of things happened to me in and after the upgrade:

• Fedora 22 appears to have shuffled around the /dev/disk/by-id names for disks in a way that broke automatic boot time importing of my ZFS pools until I imported them once by hand. I'm not entirely happy with this, but I am running an unsupported configuration.

• systemd's networkd exhibited systemd's usual regard for its users by changing which of several IP addresses would be an interface's primary IP address. Apparently my hope was naive.

  (This is where some systemd person smugly observes that the order is not documented and so I deserve whatever I get here, including random decisions from boot to boot. See 'usual regard for its users', above.)

• Fedora 22 has a broken rsyslog(d) that will casually and trivially dump core. Fortunately the code flaw is trivially fixable; it's a good thing that I know how to patch RPMs by hand. An update is coming out sometime, but apparently Fedora does not consider 'the syslog daemon dumps core' to be a high priority issue.

  (This feels like a facet of the great systemd thing about syslog.)

• Fedora 22 now spews audit system messages all over your kernel logs by default, which is especially fun if you just got rsyslog working and would like to watch your kernel logs for important anomalies. I have so far disabled most of this with 'auditctl -e 0'; my long term fix is going to be adding 'audit=0' to the kernel command line. I wish I knew what changed in Fedora 22 to cause these messages to start showing up in my configuration, but, well, who knows.

  (I also modified my rsyslog configuration to divert those messages to another file, using a very brute force method because I was angry and in a hurry.)

• I ran into a gcc 5.1.1 bug.

And then there's DNF, the Fedora 22 replacement for yum. Oh, DNF, what can I say about you.

I believe the Fedora and DNF people when they say that the internals of DNF are better than the internals of Yum. But it's equally clear to me that DNF is nowhere near as usable and polished as Yum and so it has a ton of irritations in day to day usage. My experience with DNF has it slow and balky and erratic as compared to the smooth and working Yum I'm used to, and I've been neither impressed nor enthused about the forced switch. From a user perspective, this is not an improvement, it's a whole bunch of regressions.

On top of that, it's pretty clear that no one has ever seriously used or tested the dnf 'local' plugin, which lets you keep a copy of all packages you install through DNF. I've used the equivalent Yum plugin for years so that I could roll back to older versions of packages if I needed to (ie, when a package 'improvement' has broken the new current version for me). The DNF version has a truly impressive collection of 'this thing doesn't work' bugs. I managed to get it sort of working by dint of being both fairly familiar with how this stuff works under the hood and willing to edit the DNF Python source, and even then it sometimes explodes.

(Many people may not care about this but I actually use yum quite frequently, so a balky, stalling, uninformative, and frustrating version of it is really irritating. Everything I do with DNF seems to take twice as long and be twice as irritating as it was with Yum.)

At this point some people will reasonably ask if upgrading to Fedora 22 was worth it. My current answer is 'yes, sort of, and it's not as if I have a choice here'. To run Fedora is to be on an upgrade treadmill, like it or not, and Fedora 22 does improve and modernize some things. All of this annoyance is just the price I periodically pay for running Fedora instead of any of the alternatives.

(And yes, I still prefer Fedora to Debian, Ubuntu, or FreeBSD.)

Multiple set matches with Linux's iptables 'ipset' extension

Recently, Luke Atchew approached me on Twitter to ask if I had any ideas for solving an ipset challenge. Suppose that you have a set of origin hosts, a set of destination servers, and you want to allow traffic from the origin hosts to the destination servers while blocking all other traffic. As Luke noted, most ipset examples, including mine, are about blocking access to or from a set of hosts; they don't have a matrix setup like Luke's.

One obvious option is a complex multi-rule setup, where you have a series of rules that try to block and accept more and more of the traffic that you want; for example, you can start out by blocking all traffic to '! -match-set Destinations dst'. This gets complex if you have other rules involved, though. Another option is one of ipset's more complicated set types, such as hash:net,net, but then you get into hassles populating and maintaining the set (since it's basically the cross product of your allowed source and destination hosts).

As Luke Atchew discovered while working on this, all of this complexity is unnecessary because you can match against multiple ipset sets in the same iptables rule. It is perfectly legitimate to write rules like:

iptables -A FORWARD -m set --match-set Locals src -m set --match-set Remotes dst -j ACCEPT
iptables -A FORWARD -m set ! --match-set Locals src -m set --match-set Remotes dst -j DROP

(These rules use different keys, here the source and destination IPs, but it's probably legitimate to have several --match-set's that reuse the same key. You might need a source IP to be in an 'allowed' set and not in a 'temporarily blocked' set, for example.)

An important note here is that you absolutely have to use two '-m set' arguments, one before each --match-set. If you leave the second one out you will get the somewhat obscure error message 'iptables v1.4.21: --match-set can be specified only once', which may mislead you into believing that this isn't supported at all. This is a really easy mistake to make because it sure smells like the second -m set is surplus; after all, you already told iptables you wanted to use the ipset extension here.

(I assume that the second '-m set' is causing the parser to start setting up a new internal ipset matching operation instead of accumulating more options for the first one.)

I've finally turned SELinux fully off even on my laptop

As I've mentioned before, I started out with SELinux turned on on my laptop because it's essentially a stock Fedora install and that's how Fedora defaults, and using SELinux felt virtuous. Last year I reached the end of my patience with running SELinux in enforcing mode, where it actually denies access to things; instead I switched it to permissive, where it just whines about things that it would have forbidden and then a whole complicated pile of software springs into action to tell you about these audit failures with notifications, popup dialogs and so on.

Today I gave up on that. My laptop now has SELinux disabled entirely (as my desktop machines have for years). The cause is simple: too many SELinux violations kept happening and especially too many new and different ones kept coming up. I am only willing to play whack a mole on notification alerts for so long before I stop caring entirely, and I reached that point today. The simplest and most easily reversed way to stop getting notifications about SELinux violations is to set the SELinux policy to disabled in /etc/selinux/config, so that's what I did.

It's possible that some of the problem is due to just upgrading to Fedora 22 with yum instead of, say, fedup, and perhaps it could be patched up somewhat with 'restorecon -R /'. Perhaps a wholesale reinstall would reduce it even more (at the cost of putting me through a wholesale reinstall and then figuring out how to set up my environment and my account and keys and wifi access and VPNs and so on all over again). Certainly I assume that SELinux has to work for some people on Fedora. But I no longer care. I am done with being quixotically virtuous and suffering for it.

(I originally put a rant about Fedora and SELinux here, but after thinking about it I took it out again. It's nothing I haven't said before and I can't be sure that my SELinux problems would still be there if I did absolutely everything the officially approved Fedora way. Since I'm never going to stop eg doing Fedora version updates with yum, well, that case will never apply to me.)

What I plug into my home Linux machine as far as peripherals go

While I've talked about the actual hardware of my home Linux machine, that's only part of a complete system. Today, for reasons beyond the scope of this entry, I feel like talking about everything else that goes into a complete system that you (well, I) can actually use as a desktop machine to get things done.

My current home display is a Dell U2412M and yes, I know, only one display may strike people as a little bit odd all things considered (I have dual displays at work). One part of it is that I kind of like not having an overwhelming amount of screen real estate at home so I don't try to do too much, and another part of it is that there is a whole chain of yak shaving necessary before I could actually really fit in a second display. The desire to avoid an overwhelming screen presence at home is also part of why it's only a 24" display instead of something larger.

What I'd really like here is a high DPI LCD panel (even with my concerns), but while they're starting to become available they're nowhere near what I consider a sane price just yet. I hope to not have to buy another display before high DPI display prices drop significantly.

My keyboard is the now out of production BTC-5100C mini keyboard. I love this 80-key PS/2 keyboard unreasonably; it's just right for my tastes. Maybe there's a better keyboard for me out there somewhere, but I probably won't find out for quite a while; I stockpiled some spares from eBay when I discovered that BTC had stopped producing it.

(I know that there are various mini mechanical keyswitch keyboards, which seem the most likely candidates for a replacement. Possibly a quiet version of one would be to my tastes and give me a better typing experience than the BTC-5100C, although I'm very acclimatized to its keyboard feel by now.)

My primary mouse is an old Logitech plain three-button mechanical ball mouse. Actually now that I look my current home one is a rebadged Digital mouse that we got at one point with some Digital PCs (which makes it something like fifteen years old; it is of course a PS/2 mouse), but it's a Logitech under the label. I am strongly attached to real three button mice, for good reasons. These days I also have a secondary mouse purely for its scroll wheel; the current one is a generic Lenovo USB one but basically any of them would do.

What I would like is a three button USB mouse with a scroll wheel on the left side where my (right hand) thumb would naturally rest; that would let me get rid of the hack of a second mouse. I don't expect to find this any time soon. In the mean time, I have a bunch of old PS/2 three button mice stockpiled for when my current ones wear out.

The weak point in both my keyboard and mouse plans are the need for PS/2 connectors. This means that I should stock up on some PS/2 keyboard and mouse to USB converters while they're still available, assuming that they still are. Otherwise, well, someday people are going to stop making motherboards with PS/2 ports and I will be in trouble.

(I have given up on what was once a stubborn opposition to USB for keyboards and mice. They work okay and they're basically inevitable at this point.)

I listen to random music (and YouTube videos) through some inexpensive computer speakers that are not particularly great and that I'm not sure I like the sound of (also, they buzz quietly, which is especially annoying to me because I like to listen to music at low volume, where the music doesn't drown out the buzz). I'd like decent speakers and a better way to switch in headphones if I want to listen that way (I have decent headphones but rarely use them because getting them out and plugged in and adjusting the volume and so on is too much of a pain as things are wired up today). Unfortunately there are a daunting number of choices in computer audio gear and so I never get anywhere with this.

(The deluxe solution would be an outboard USB DAC plus a headphone amp that can also drive typical powered computer speakers. I understand that this setup can sound very nice, especially with decent headphones.)

I don't have but should get a USB 3.0 external disk enclosure for backups (and at least one drive to go in it). My current solution for backups is sufficiently awkward that I only back up my home machine once in a blue moon (such as when I've been alarmed because one side of my RAID mirror is reporting problems).

My work machine has essentially the same set of peripherals (same keyboard, same sort of primary mouse, etc). The speakers are different, of course, because I basically wound up with some random speakers we had sitting around. I'm hoping they never die, because I'll let you imagine how likely it would be to talk a university into buying nice modern speakers for a sysadmin machine.

(The obvious plan is to take my current home speakers in to work if I ever get better ones for home.)

Linux evolution through de facto quiet standards

In my entry on why I was interested in moving to ext4, I said:

As time goes by, more and more software is likely to assume sub-second file timestamps basically by default (because the authors have never run it on a system without them) and not work quite right in various ways. I can fight a slow battle against what is effectively a new standard of sub-second file timestamps, [...]

Some people might bristle at the idea that we will drift into a world where sub-second file timestamps are a standard not because people sat down and thought about it but because it just kind of happened. I have pretty much the opposite reaction.

Linux (and Unix in general) has always evolved in part through a familiar process where common environments have something (such as sub-second file timestamps), people write code that tacitly assumes this thing because that's how people develop, and then this thing becomes basically mandatory in that if you try to use or push an environment without it, you wind up having a painful time. This is not a bad thing (or at least almost never a bad thing). Instead it is exactly how a de facto standard forms and how you want them to form. In many ways the best standards are the ones that arise organically because they're what everyone is doing and what everyone already has (for suitable values of 'everyone'). Arguably it's how standardization is supposed to work, where would-be standards are tested in the marketplace of code and either found useful or found wanting.

(Once you have a de facto standard like this, you can get some standards people to come and write it down for you if you want an actual formal document about it.)

If sub-second file timestamps become a real de facto standard for Linux (or even for Unix), it won't be anything exceptional. It will just be another bit in a long line of Linux (and Unix) evolution through what everyone just does.

(Where de facto standards like this tend to run into problems is situations where the world changes. After all, once upon a time everything was a 32-bit Vax and people wrote code accordingly. Then the situation changed repeatedly and, well, pain ensued.)

By the way, not everyone is happy with de facto standards forming this way. As to why, well, one way to put it is that Gnome and DBus and so on are currently de facto Linux standards and systemd may well be on the way to being one (the systemd people probably would like that). The possibility of de facto standards forming tacitly is undoubtedly one reason some people fought so hard against systemd in Debian.

Converting filesystems from ext3 to ext4, and concerns attached to it (plus bad news for me)

Yesterday I covered why I basically need to move from ext3 to ext4 and I said that the mechanisms of this transition raise a few issues. So let's talk about them, and in the process I'll wind up with disappointing news for my own case. Actually, let's lead with the disappointing news:

An ext3 filesystem converted to ext4 almost certainly won't support ext4's nanosecond file timestamps; it will only have ext3 one-second ones.

On the surface the mechanics of converting from ext3 to ext4 are simple and relatively straightforward, with two levels. The first level is simply to mount filesystems as ext4 instead of ext3. This enables backwards-compatible filesystem features and is fully reversible (at least according to the sources I've read). As a result it ought to be as safe as ext4 itself, and these days that's pretty safe. However this only gives you features like delayed allocation (which some people consider a non-feature for complex reasons).

(Checksumming the journal may be one of the things you get here; it's not clear to me.)

To fully convert a filesystem to ext4, one must also use tune2fs to enable the ext4-specific filesystem features that affect the on disk format of the filesystem. Various sources (eg the ext4 wiki) say that this is:

tune2fs -O extents,uninit_bg,dir_index /dev/<WHAT>

(In theory dir_index is also an ext3 feature, but on the other hand at least some of my ext3 filesystems don't have it. They may have started out as ext2 filesystems.)

However, this is not the full set of feature differences between ext3 and ext4 today. The Fedora 21 /etc/mke2fs.conf adds huge_file, flex_bg, dir_nlink, and extra_isize, and the tune2fs manpage says that all of them can be set on an existing filesystem (although flex_bg and extra_isize probably have no meaningful effect). Per the tune2fs manpage and various instructions on ext3 to ext4 conversions, setting uninit_bg requires an e2fsck and setting dir_index is helped by doing an e2fsck -D.

Now we get to my concerns. The first concern is straightforward; at least some 'how to convert from ext3 to ext4' sources contain prominent 'back up your data just in case' warnings. These sources are also relatively old ones and I suspect that they date from the days when ext4 and this process was new and thus uncertain. Notably, the ext4 wiki itself doesn't have any such cautions these days. Still, it's a bit alarming.

The second concern is whether this is actually going to do me any good. My entire purpose for doing this is to get support for sub-second file timestamps, because that's basically the only user visible ext3 versus ext4 difference (and software is noticing). The existing documentation is not clear on whether ext4 automatically starts doing this on an existing ext3 filesystem, if you have to add the extra_isize feature, or if a converted ext3 filesystem simply can't do this because its existing (and future) inodes are too small to hold the extra data.

Unfortunately the news here is not good. Additional sources (this description of timestamps in ext4, this SA answer) say that the high resolution timestamps definitely go in the extra space in normal ext4 inodes, space that my ext3 filesystem inodes just don't have (ext3 defaults to 128 byte inodes, ext4 to 256 byte ones). Converting a filesystem from ext3 to ext4 does not enlarge its inodes, which means that a normal ext3 filesystem converted to ext4 will never support high resolution timestamps.

Given this, there's not much point in putting myself through this in-place conversion effort (in fact converting my filesystems would be silently misleading). The only way I'd ever get better timestamps would be to make new ext4 filesystems from scratch, copy all the current data into them, and then replace my existing filesystem mounts with mounts of the new ext4 filesystems. Among other things, this is a pain in the rear.

(It's achievable, at least in theory, as my LVM pool has plenty of unused space. It would mean that I'd only 'convert' a couple of filesystems, the ones most likely to run into high resolution file timestamp issues.)

PS: You can see how big your ext3 inodes are with 'tune2fs -l /dev/<WHAT>'; it's reported near the end. This will also let you see what features are set on your filesystem (and thus let you know that your ext3 filesystems are without dir_index).

Why I'm interested in converting my ext3 filesystems to ext4

My home machine has a sufficiently old set of filesystems that many of my actively used filesystems are still ext3, not ext4, including both my home directory and where I keep code. Normally this isn't something that I particularly think or worry about; it's not like ext4 is a particularly radical advance from ext3 (certainly not the same sort of jump that was ext2 to ext3, where you got fast crash recovery). As a sysadmin I'm generally cautious with filesystem choices anyways (at least when I'm not being radical); I used ext2 over ext3 for years after the latter came out, for example, on the principle that I'd let other people find the problems.

It turns out that there is one important thing that ext4 has and ext3 does not: ext4 has sub-second file timestamps, while ext3 only does timestamps to the nearest second. Modern machines are fast enough that nearest second timestamps are increasingly not really good enough when building software or otherwise doing things that care about relative file timestamps and 'is X more recent than Y'. Oh, sure, it works most of the time, but every so often things go wrong or you find assumptions buried in other people's software.

Most people don't notice these things because most people are now using filesystems that support sub-second file timestamps (which is almost all modern Linux filesystems). What this tells me is that I'm increasingly operating in an unusual and effectively unsupported environment by continuing to use ext3. As time goes by, more and more software is likely to assume sub-second file timestamps basically by default (because the authors have never run it on a system without them) and not work quite right in various ways. I can fight a slow battle against what is effectively a new standard of sub-second file timestamps, or I can give in and convert my ext3 filesystems to ext4. It's not like ext4 is exactly a new filesystem these days, after all (Wikipedia dates it to 2008).

The mechanics of this conversion raise a few issues, but that's something for another entry.