Wandering Thoughts archives

My early impressions of Fedora 22, especially of DNF

I recently updated first my office laptop (which runs a relatively stock Cinnamon environment) and my office workstation (which runs my custom setup) to Fedora 22, both via my usual means of a yum-based upgrade instead of the officially supported FedUp mechanism. I feel kind of ambivalent about the results of this.

On the one hand, the upgrade was smooth in both cases and everything in both of my environments basically worked from the start. This is not always the case, especially in my custom setup; I'm used to having to fiddle things around after Fedora version upgrades in order to get audio or automatic removable media mounting or whatever working again. Instead everything pretty much just went, and nothing changed in my Cinnamon environment.

On the other hand, how can I say this gently: I have not been really impressed with Fedora 22's quality control. A number of things happened to me in and after the upgrade:

• Fedora 22 appears to have shuffled around the /dev/disk/by-id names for disks in a way that broke automatic boot time importing of my ZFS pools until I imported them once by hand. I'm not entirely happy with this, but I am running an unsupported configuration.

• systemd's networkd exhibited systemd's usual regard for its users by changing which of several IP addresses would be an interface's primary IP address. Apparently my hope was naive.

  (This is where some systemd person smugly observes that the order is not documented and so I deserve whatever I get here, including random decisions from boot to boot. See 'usual regard for its users', above.)

• Fedora 22 has a broken rsyslog(d) that will casually and trivially dump core. Fortunately the code flaw is trivially fixable; it's a good thing that I know how to patch RPMs by hand. An update is coming out sometime, but apparently Fedora does not consider 'the syslog daemon dumps core' to be a high priority issue.

  (This feels like a facet of the great systemd thing about syslog.)

• Fedora 22 now spews audit system messages all over your kernel logs by default, which is especially fun if you just got rsyslog working and would like to watch your kernel logs for important anomalies. I have so far disabled most of this with 'auditctl -e 0'; my long term fix is going to be adding 'audit=0' to the kernel command line. I wish I knew what changed in Fedora 22 to cause these messages to start showing up in my configuration, but, well, who knows.

  (I also modified my rsyslog configuration to divert those messages to another file, using a very brute force method because I was angry and in a hurry.)

• I ran into a gcc 5.1.1 bug.

And then there's DNF, the Fedora 22 replacement for yum. Oh, DNF, what can I say about you.

I believe the Fedora and DNF people when they say that the internals of DNF are better than the internals of Yum. But it's equally clear to me that DNF is nowhere near as usable and polished as Yum and so it has a ton of irritations in day to day usage. My experience with DNF has it slow and balky and erratic as compared to the smooth and working Yum I'm used to, and I've been neither impressed nor enthused about the forced switch. From a user perspective, this is not an improvement, it's a whole bunch of regressions.

On top of that, it's pretty clear that no one has ever seriously used or tested the dnf 'local' plugin, which lets you keep a copy of all packages you install through DNF. I've used the equivalent Yum plugin for years so that I could roll back to older versions of packages if I needed to (ie, when a package 'improvement' has broken the new current version for me). The DNF version has a truly impressive collection of 'this thing doesn't work' bugs. I managed to get it sort of working by dint of being both fairly familiar with how this stuff works under the hood and willing to edit the DNF Python source, and even then it sometimes explodes.

(Many people may not care about this but I actually use yum quite frequently, so a balky, stalling, uninformative, and frustrating version of it is really irritating. Everything I do with DNF seems to take twice as long and be twice as irritating as it was with Yum.)

At this point some people will reasonably ask if upgrading to Fedora 22 was worth it. My current answer is 'yes, sort of, and it's not as if I have a choice here'. To run Fedora is to be on an upgrade treadmill, like it or not, and Fedora 22 does improve and modernize some things. All of this annoyance is just the price I periodically pay for running Fedora instead of any of the alternatives.

(And yes, I still prefer Fedora to Debian, Ubuntu, or FreeBSD.)

Multiple set matches with Linux's iptables 'ipset' extension

Recently, Luke Atchew approached me on Twitter to ask if I had any ideas for solving an ipset challenge. Suppose that you have a set of origin hosts, a set of destination servers, and you want to allow traffic from the origin hosts to the destination servers while blocking all other traffic. As Luke noted, most ipset examples, including mine, are about blocking access to or from a set of hosts; they don't have a matrix setup like Luke's.

One obvious option is a complex multi-rule setup, where you have a series of rules that try to block and accept more and more of the traffic that you want; for example, you can start out by blocking all traffic to '! -match-set Destinations dst'. This gets complex if you have other rules involved, though. Another option is one of ipset's more complicated set types, such as hash:net,net, but then you get into hassles populating and maintaining the set (since it's basically the cross product of your allowed source and destination hosts).

As Luke Atchew discovered while working on this, all of this complexity is unnecessary because you can match against multiple ipset sets in the same iptables rule. It is perfectly legitimate to write rules like:

iptables -A FORWARD -m set --match-set Locals src -m set --match-set Remotes dst -j ACCEPT
iptables -A FORWARD -m set ! --match-set Locals src -m set --match-set Remotes dst -j DROP

(These rules use different keys, here the source and destination IPs, but it's probably legitimate to have several --match-set's that reuse the same key. You might need a source IP to be in an 'allowed' set and not in a 'temporarily blocked' set, for example.)

An important note here is that you absolutely have to use two '-m set' arguments, one before each --match-set. If you leave the second one out you will get the somewhat obscure error message 'iptables v1.4.21: --match-set can be specified only once', which may mislead you into believing that this isn't supported at all. This is a really easy mistake to make because it sure smells like the second -m set is surplus; after all, you already told iptables you wanted to use the ipset extension here.

(I assume that the second '-m set' is causing the parser to start setting up a new internal ipset matching operation instead of accumulating more options for the first one.)

I've finally turned SELinux fully off even on my laptop

As I've mentioned before, I started out with SELinux turned on on my laptop because it's essentially a stock Fedora install and that's how Fedora defaults, and using SELinux felt virtuous. Last year I reached the end of my patience with running SELinux in enforcing mode, where it actually denies access to things; instead I switched it to permissive, where it just whines about things that it would have forbidden and then a whole complicated pile of software springs into action to tell you about these audit failures with notifications, popup dialogs and so on.

Today I gave up on that. My laptop now has SELinux disabled entirely (as my desktop machines have for years). The cause is simple: too many SELinux violations kept happening and especially too many new and different ones kept coming up. I am only willing to play whack a mole on notification alerts for so long before I stop caring entirely, and I reached that point today. The simplest and most easily reversed way to stop getting notifications about SELinux violations is to set the SELinux policy to disabled in /etc/selinux/config, so that's what I did.

It's possible that some of the problem is due to just upgrading to Fedora 22 with yum instead of, say, fedup, and perhaps it could be patched up somewhat with 'restorecon -R /'. Perhaps a wholesale reinstall would reduce it even more (at the cost of putting me through a wholesale reinstall and then figuring out how to set up my environment and my account and keys and wifi access and VPNs and so on all over again). Certainly I assume that SELinux has to work for some people on Fedora. But I no longer care. I am done with being quixotically virtuous and suffering for it.

(I originally put a rant about Fedora and SELinux here, but after thinking about it I took it out again. It's nothing I haven't said before and I can't be sure that my SELinux problems would still be there if I did absolutely everything the officially approved Fedora way. Since I'm never going to stop eg doing Fedora version updates with yum, well, that case will never apply to me.)

What I plug into my home Linux machine as far as peripherals go

While I've talked about the actual hardware of my home Linux machine, that's only part of a complete system. Today, for reasons beyond the scope of this entry, I feel like talking about everything else that goes into a complete system that you (well, I) can actually use as a desktop machine to get things done.

My current home display is a Dell U2412M and yes, I know, only one display may strike people as a little bit odd all things considered (I have dual displays at work). One part of it is that I kind of like not having an overwhelming amount of screen real estate at home so I don't try to do too much, and another part of it is that there is a whole chain of yak shaving necessary before I could actually really fit in a second display. The desire to avoid an overwhelming screen presence at home is also part of why it's only a 24" display instead of something larger.

What I'd really like here is a high DPI LCD panel (even with my concerns), but while they're starting to become available they're nowhere near what I consider a sane price just yet. I hope to not have to buy another display before high DPI display prices drop significantly.

My keyboard is the now out of production BTC-5100C mini keyboard. I love this 80-key PS/2 keyboard unreasonably; it's just right for my tastes. Maybe there's a better keyboard for me out there somewhere, but I probably won't find out for quite a while; I stockpiled some spares from eBay when I discovered that BTC had stopped producing it.

(I know that there are various mini mechanical keyswitch keyboards, which seem the most likely candidates for a replacement. Possibly a quiet version of one would be to my tastes and give me a better typing experience than the BTC-5100C, although I'm very acclimatized to its keyboard feel by now.)

My primary mouse is an old Logitech plain three-button mechanical ball mouse. Actually now that I look my current home one is a rebadged Digital mouse that we got at one point with some Digital PCs (which makes it something like fifteen years old; it is of course a PS/2 mouse), but it's a Logitech under the label. I am strongly attached to real three button mice, for good reasons. These days I also have a secondary mouse purely for its scroll wheel; the current one is a generic Lenovo USB one but basically any of them would do.

What I would like is a three button USB mouse with a scroll wheel on the left side where my (right hand) thumb would naturally rest; that would let me get rid of the hack of a second mouse. I don't expect to find this any time soon. In the mean time, I have a bunch of old PS/2 three button mice stockpiled for when my current ones wear out.

The weak point in both my keyboard and mouse plans are the need for PS/2 connectors. This means that I should stock up on some PS/2 keyboard and mouse to USB converters while they're still available, assuming that they still are. Otherwise, well, someday people are going to stop making motherboards with PS/2 ports and I will be in trouble.

(I have given up on what was once a stubborn opposition to USB for keyboards and mice. They work okay and they're basically inevitable at this point.)

I listen to random music (and YouTube videos) through some inexpensive computer speakers that are not particularly great and that I'm not sure I like the sound of (also, they buzz quietly, which is especially annoying to me because I like to listen to music at low volume, where the music doesn't drown out the buzz). I'd like decent speakers and a better way to switch in headphones if I want to listen that way (I have decent headphones but rarely use them because getting them out and plugged in and adjusting the volume and so on is too much of a pain as things are wired up today). Unfortunately there are a daunting number of choices in computer audio gear and so I never get anywhere with this.

(The deluxe solution would be an outboard USB DAC plus a headphone amp that can also drive typical powered computer speakers. I understand that this setup can sound very nice, especially with decent headphones.)

I don't have but should get a USB 3.0 external disk enclosure for backups (and at least one drive to go in it). My current solution for backups is sufficiently awkward that I only back up my home machine once in a blue moon (such as when I've been alarmed because one side of my RAID mirror is reporting problems).

My work machine has essentially the same set of peripherals (same keyboard, same sort of primary mouse, etc). The speakers are different, of course, because I basically wound up with some random speakers we had sitting around. I'm hoping they never die, because I'll let you imagine how likely it would be to talk a university into buying nice modern speakers for a sysadmin machine.

(The obvious plan is to take my current home speakers in to work if I ever get better ones for home.)

Linux evolution through de facto quiet standards

In my entry on why I was interested in moving to ext4, I said:

As time goes by, more and more software is likely to assume sub-second file timestamps basically by default (because the authors have never run it on a system without them) and not work quite right in various ways. I can fight a slow battle against what is effectively a new standard of sub-second file timestamps, [...]

Some people might bristle at the idea that we will drift into a world where sub-second file timestamps are a standard not because people sat down and thought about it but because it just kind of happened. I have pretty much the opposite reaction.

Linux (and Unix in general) has always evolved in part through a familiar process where common environments have something (such as sub-second file timestamps), people write code that tacitly assumes this thing because that's how people develop, and then this thing becomes basically mandatory in that if you try to use or push an environment without it, you wind up having a painful time. This is not a bad thing (or at least almost never a bad thing). Instead it is exactly how a de facto standard forms and how you want them to form. In many ways the best standards are the ones that arise organically because they're what everyone is doing and what everyone already has (for suitable values of 'everyone'). Arguably it's how standardization is supposed to work, where would-be standards are tested in the marketplace of code and either found useful or found wanting.

(Once you have a de facto standard like this, you can get some standards people to come and write it down for you if you want an actual formal document about it.)

If sub-second file timestamps become a real de facto standard for Linux (or even for Unix), it won't be anything exceptional. It will just be another bit in a long line of Linux (and Unix) evolution through what everyone just does.

(Where de facto standards like this tend to run into problems is situations where the world changes. After all, once upon a time everything was a 32-bit Vax and people wrote code accordingly. Then the situation changed repeatedly and, well, pain ensued.)

By the way, not everyone is happy with de facto standards forming this way. As to why, well, one way to put it is that Gnome and DBus and so on are currently de facto Linux standards and systemd may well be on the way to being one (the systemd people probably would like that). The possibility of de facto standards forming tacitly is undoubtedly one reason some people fought so hard against systemd in Debian.