Wandering Thoughts archives

How to talk to a local IPMI under FreeBSD 14

Much like Linux and OpenBSD, FreeBSD is able to talk to a local IPMI using the ipmi kernel driver (or device, if you prefer). This is imprecise although widely understood terminology; in more precise terms, FreeBSD can talk to a machine's BMC (Baseboard Management Controller) that implements the IPMI specification in various ways which you seem to normally not need to care about (for information on 'KCS' and 'SMIC', see the "System Interfaces" section of OpenBSD's ipmi(4)).

Unlike in OpenBSD (covered earlier), the stock FreeBSD 14 kernel appears to report no messages if your machine has an IPMI interface but the driver hasn't been enabled in the kernel. To see if your machine has an IPMI interface that FreeBSD can talk to, you can temporarily load the ipmi module with 'kldload ipmi'. If this succeeds, you will see kernel messages that might look like this:

ipmi0: <IPMI System Interface> port 0xca8,0xcac irq 10 on acpi0
ipmi0: KCS mode found at io 0xca8 on acpi
ipmi0: IPMI device rev. 1, firmware rev. 7.10, version 2.0, device support mask 0xdf
ipmi0: Number of channels 2
ipmi0: Attached watchdog
ipmi0: Establishing power cycle handler

(On the one Dell server I've tried this on so far, the ipmi(4) driver found the IPMI without any special parameters.)

At this point you should have a /dev/ipmi0 device and you can 'pkg install ipmitool' and talk to your IPMI. To make this permanent, you edit /boot/loader.conf to load the driver on boot, by adding:

ipmi_load="YES"

While you're there, you may also want to load the coretemp(4) module or perhaps amdtemp(4). After updating loader.conf, you need to reboot to make it take full effect, although since you can kldload everything before then I don't think there's a rush.

In FreeBSD, IPMI sensor information isn't visible in sysctl (although information from coretemp or amdtemp is). You'll need ipmitool or another suitable program to query it. You can also use ipmitool to configure the basics of the IPMI's networking and set the IPMI administrator's password to something you know, as opposed to whatever unique value the machine's vendor set it to, which you may or may not have convenient access to.

(As far as I can tell, ipmitool works the same on FreeBSD as it does on Linux, so if you have existing scripts and so on that use it for collecting data on your Linux hosts (as we do), they will probably be easy to make work on any FreeBSD machines you add.)

What a POSIX shell has to do with $PWD

It's reasonably well known about Unix people that '$PWD' is a shell variable with the name of the current working directory. Well, sort of, because sometimes $PWD isn't right or isn't even set (all of this is part of the broader subject of shells and the current directory). Until recently, I hadn't looked up what POSIX has to say about $PWD, and when I did I was surprised, partly because I didn't expect POSIX to say anything about it.

(Until I looked it up, I had the vague impression that $PWD was a common but non-POSIX Bourne shell thing.)

What POSIX has to say is in 2.5.3 Shell Variables part of the overall description of the POSIX shell. To put my own summary on what POSIX says, the shell creates and maintains $PWD in basically all circumstances, and is obliged to update $PWD when it does a 'cd', even in shell scripts. The only case where $PWD's value isn't specified in the shell environment is if you don't have access permissions for the current directory for some reason.

(As far as I can tell, the complicated POSIX wording boils down to that if you start the shell with a correct $PWD that uses symbolic links (eg '/u/cks' instead of '/h/281/cks'), the shell is allowed to update that to the post-symlink 'physical' version but doesn't have to. See how 'pwd -P' is described.)

However, $PWD is not necessarily correct when you're running a program written in C, because POSIX chdir() doesn't seem to be required to update $PWD for you (although it's a bit confusing, since Environment Variables seems to imply that POSIX utilities are entitled to believe $PWD is correct if it's in the environment). In fact I don't think that the POSIX shell is always obliged to export $PWD into the environment, which is why I called it a shell variable instead of an environment variable. I believe most actual Bourne shell implementations do always export $PWD, even if they're started in an environment with it undefined (where I believe POSIX allows it to not be exported).

(Bash, Dash, and FreeBSD's Almquist shell all allow $PWD to be unexported, although keeping it that way may be tricky in Dash and FreeBSD sh, which appear to re-export it any time you do a 'cd'.)

The upshort of this is that in a modern environment where /bin/sh is a POSIX shell, $PWD will almost always be correct. It pretty much has to be correct in your POSIX shell sessions and in your POSIX shell scripts. POSIX-compatible shells like Bash will keep it correct even in their more expansive modes, and non-Bourne shells have a strong motive to go with the show because people expect $PWD to work and be correct.

(However, this leaves me mystified about what the problem was in my specific circumstance this time around, since I'd expect $PWD to have gotten set correctly when my /bin/sh based script used 'cd'.)

FreeBSD's 'root on ZFS' default appeals to me for an odd reason

For reasons beyond the scope of this entry, we're probably going to take a look at FreeBSD as an alternative to OpenBSD for some of our uses of the latter. This got me to grab a 14.1 ISO image and try a quick install on a spare virtual machine (I keep spare VMs around for just such occasions). This caused me to discover that modern FreeBSD defaults to using ZFS for its root filesystem (although I didn't do this on my VM test install, because my VM has less than the recommended RAM for ZFS). FreeBSD using ZFS for its root filesystem makes me happy, but probably not quite for the reasons you're expecting.

Certainly, I like ZFS in general and I think it has a bunch of nice properties, even for a root filesystem. You get checksums for reliability, compression, the ability to easily add sub-filesystems if you want to limit the amount of space something can use (we have usage cases for this, but that's another entry), and so on. But these aren't what make me happy for it as a root filesystem on FreeBSD. The really nice thing about root on ZFS on FreeBSD for me is the easy mirroring.

A traditional thing with all of our non-Linux installs is that they don't have mirrored system disks. We've made some stabs at it in the past but at the time we found it complex and not clearly compelling, perhaps partly because we didn't have experience with their software mirroring systems. Well, we have a lot of experience with mirroring ZFS vdevs and it's trivial to set ZFS mirroring up after the fact or to revert back from a mirrored setup to a single-disk setup. So while we might not bother going through the hassles of learning a FreeBSD-specific software mirroring system, we're pretty likely to use ZFS mirroring on any production FreeBSD machines. And that will be a good thing for our FreeBSD machines in general.

(Using ZFS for the root filesystem also eliminates any chance that the server will ever stall in boot asking us to approve a fsck, something that has happened to our OpenBSD machines under rare circumstances.)

I'm also personally pleased to see a fully supported 'root on ZFS' in anything. My impression is that FreeBSD is reasonably well used, so their choice of ZFS for the default root filesystem setup may even be exposing a reasonable number of people to (Open)ZFS and its collection of nice things.

PS: our OpenBSD machines come in pairs and we've had very good luck with their root drives, or we might have looked into the OpenBSD bioctl(8) software mirroring system and how you install to a mirror.

Host names in syslog messages may not be quite what you expect

Over on the Fediverse, I said something:

It has been '0' days since I (re)discovered that the claimed hostname in syslog messages can be utter junk, and you may be going to live a fun life if you use it for anything much.

Suppose that on your central syslog server you see a syslog line of the form:

[...] alkyone exim[864974]: no host name found for IP address 115.187.17.119

You might reasonably assume that the host name 'alkyone' comes from the central syslog daemon knowing the host name of the host that sent the syslog message to it. Unfortunately, this is not what actually happens. As covered in places like RFC 5424 section 6.2.4 (or RFC 3164 section 4.1.2 for the nominal 'BSD' syslog format, which seem to not actually be what BSD used), syslog messages carry an embedded hostname in them. This hostname is generated by the machine that originated the message, and the machine can put anything it wants to in there. And generally, your syslog daemon (and the log format it's using) will write this hostname into the logs and otherwise use it if you ask for the message's 'hostname'.

(Rsyslog and probably other syslog daemons can create per-host message files on your central syslog server, which can cause you to want a hostname for each message.)

The intent of this embedded hostname is noble; it's there so you can have syslog relays (which may happen accidentally), where the originating system sends its messages to host A and host A relays them to host B, and B records the hostname as the originating system, not host A. Unfortunately, in practice all sorts of things can go wrong, including a quite fun one.

The first thing that can go wrong is systems that have a different view of their hostname than you do. On Unix systems, the normal syslog hostname traditionally comes from whatever the general host name is set to, which isn't necessarily a fully qualified domain name and doesn't necessarily match what its IP address is (you can change the IP address of a system but forget to update its hostname). Some embedded systems will have an internally set host name instead of trying to deduce it from DNS lookups of whatever IP they have, which can cause them to use syslog hostnames like 'idrac-<asset-tag>' (for the BMC of a Dell server with that particular asset tag).

The most fun case is an interaction with a long-standing syslog feature (that I think is often disabled today):

<host> /bsd: arp: attempt to overwrite entry for [...]
last message repeated 2 times

You'll notice that the second message doesn't say '<host> last message repeated ...'. This is achieved with the extremely brute force method of setting the hostname in the message to 'last'. If your central syslog server then attempts to set up per-host syslog logs, you will wind up with a 'last' host (with extremely uninteresting logs).

Also, if people send not quite random garbage to your syslog server's listening network ports (perhaps because they are a vulnerability scanner or nmap or the like), your syslog daemon and your logs can wind up seeing all sorts of weird junk as the nominal hostname. The syslog message format is deliberately relatively liberal and syslog servers have traditionally been even more liberal about interpreting things that arrived on it, on the sensible grounds that it's usually better to record everything you get just in case.

Sidebar: Hostnames in syslog messages appear to be new-ish

In 4.2 BSD, the syslog daemon was part of the sendmail source code, and sendmail/aux/syslog.c doesn't get the hostname from the message but instead from the IP address it came from. I think this continues right through 4.4 BSD if I'm reading the code right. RFC 3164 dates from 2001, so presumably people augmented the syslog format some time before then.

Interestingly, RFC 3164 specifically says that the host name in the message must not include the domain name. I suspect that even at the time this was widely ignored in practice for good operational reasons.