Things that I have repositories for on Github

This is a quick index of everything interesting that I have on Github in my account.

• MXiostat: siebenmann/mxiostat , a better iostat for Linux.

• PyMilterTools: siebenmann/python-milter-tools, a pure Python implementation of the Sendmail Milter protocol.

• python-netblock, a Python module for sets of IP address ranges (and a module for sets of integer ranges in general). Comes with a bonus netblock calculator.

• portnanny, a powerful inetd-like frontend for a single TCP service, with a great deal of filtering power.

• DWiki: siebenmann/dwiki, you're soaking in it. The software that runs this thing. See also DWikiCode.
• siebenmann/cspace-skeleton, the DWiki templates and utility files that are used to set up, lay out, and style WanderingThoughts and CSpace (to the extent that they have any style). This may be a useful illustration of template and renderer tricks, although they aren't well documented. They go on top of the templates and so on in the main DWiki codebase.

• cks-dtrace, an assortment of DTrace scripts that are currently focused on NFS v3 server, ZFS, and iSCSI initiator stuff. I used (some of) them to diagnose our fileserver performance issues.

• call.go, essentially a relatively basic netcat-like program written in Go. It will talk over a wide variety of socket types, including Unix domain sockets, and it can act as a TLS/SSL client.

• smtpd is a starting Go implementation of the server side of SMTP (net/smtp basically only handles the client side). It doesn't handle high-level issues like what addresses should be accepted and rejected; its purpose is simply to handle the grunt work of a reasonably RFC compliant but cautious SMTP server, taking care of things like proper command sequencing and TLS. It comes with a 'sinkhole' SMTP server that accepts most everything and (at most) just writes it to disk; this is suitable for things such as catching spam. Of course the SMTP server also serves as an example of how to use the package.

• sinksmtp is a vastly over-complicated 'sinkhole' SMTP server that at most just accepts email and writes it to disk; this is suitable for things such as catching spam. It uses smtpd as the underlying SMTP server engine. It has plenty of features that are designed to basically limit what it accepts so you get more interesting spam and less boring spam.

• ffox-remote is lightweight remote control for Unix Firefox (see also). It may also serve as an example of doing low level X things in Go.

• go-kstat is a Go package to read Solaris (Illumos, OmniOS, etc) kstat(s), so Go programs can have access to all of those tasty kernel statistics.

• netvolmon is a Go program to report network device utilization on Linux and Solaris. It has various crazy features and some sensible ones, and replaces my Linux script and Solaris script for doing this. Unsurprisingly, it uses go-kstat for access to network bandwidth information on Solaris; on Linux it reads /proc/net/dev.

• exim-attachment-logger is the Python code we use for our logging of MIME attachment types from Exim.

Repos that are my versions of other people's programs

All of these are likely to be rebased periodically.

• dmenu-cks is my hacks on top of the official version of dmenu.

• liferea-cks is my hacks on top of the official version of the liferea syndication feed reader.

• sam is my hacks on top of the updated, Xft-enabled version of sam.

mxiostat: better accurate Linux disk IO statistics

mxiostat is the modern and better version of xiostat. I announced it here with more information.

Because this is 2011 instead of 2006, it is hosted on github as siebenmann/mxiostat.

(This page is kind of a placeholder.)

xiostat: accurate Linux disk IO statistics

Update, September 21 2011: xiostat is now obsolete. Please use MXiostat instead. The remainder of this page is retained for historical purposes.

The Linux iostat command turns out to not report completely accurate and complete IO statistics (covered in more detail here). xiostat.py is the replacement program I wrote to give us a faithful recounting of the actual kernel disk IO statistics. It works on 2.6 kernels and Red Hat's 2.4 kernels that have disk IO stats added (such as Red Hat Enterprise Linux 3).

So that's: xiostat.py

Usage: xiostat.py [-q] [-c COUNT] [DEV [DELAY]]

Since the default device to report on is sde1 (for peculiar local reasons), the DEV argument is in practice mandatory. -q omits the field headers that are printed every 24 lines or so, -c is how many iterations to stop after (by default xiostat.py runs forever), and DELAY is how many seconds to delay between each iteration (default 1).

Feedback, fixes, etc, can be directed to ChrisSiebenmann.

Note that one significant difference between xiostat and iostat is that xiostat only reports on a single device at a time. While this could be fixed with more work (that I don't have time for right now), right now if you want to monitor multiple devices at the same time you need to run multiple copies of xiostat.

It's know to work on 2.6 kernels (and thus Debian Sarge, Fedora Core 2+, etc) and Red Hat Enterprise 4; it should work on RHEL 3 as well, but I don't have a machine to test on. It will not work on 2.4 kernels unless they have the Red Hat disk statistics patch.

xiostat fields

Xiostat prints almost all fields in amount per second, regardless of how long the DELAY is set for. Time based fields are printed in milliseconds.

Field	What
act	Instantaneous count of outstanding IO requests right now (not a per second field)
rio	Read requests completed.
rmerge	Read requests merged into existing requests
rsect	Read sectors submitted
rwait	Average time for read requests to complete
rgrp	Average sectors per read request
w*	As for r* fields, but for writes instead of reads.
agrp	Average sectors per request (across both read and write)
aveq	Average queue size, ie the average number of outstanding IO requests
await	Average time for all requests to complete (across both read and write)
util	Percentage of the time that there was at least one IO request pending.

For fine details, you should read my writeup of what information the kernel makes available.

A note about the source code

In the process of the work that caused me to write xiostat.py, we wound up adding some extra kernel statistics to monitor actual device service and activity times (to a Red Hat Enterprise 3 kernel). The remnants of code to handle these additional fields are still in the current xiostat.py, because I haven't had the energy to clean it up and validate the changes. Hence mysterious mentions of fields like 'rduse' and 'waveq'.

There is also some experimental code to parse /etc/fstab and the 2.4 LVM information to try to let people specify filesystems instead of devices. I don't believe this works well on 2.6, so you don't want to use it.

Copyright

GPL. It's not explicitly labeled as such in the code, due to lack of time et al. (Someday I will fix this, but not today.)

ljf: a command line LiveJournal friendslist and friends groups fiddler

ljf.py is a comment line Python LiveJournal client for maintaining your friends groups and friends list, with a deliberately Unixy feel. If this does not sound attractive, it is probably not for you.

This is an alpha release, which is to say that it works for me and does what I'm interested in, but it has huge gaps of functionality that would need to be fixed to make it a general tool. The biggest issue is that it is geared towards someone who uses friends groups as reading groups, not for permissions. (Specifically it assumes that you usually want each friend to be in one and only one friends group.)

Ljf is tested on Linux. It is written in pure Python and so is theoretically portable anywhere that normal Python runs. In practice it wants to read a file in your $HOME, and I have no idea how well the approach I used would work on Windows.

Documentation is in a pile of comments at the start of ljf.py.

My collection of little scripts

Like most every system administrator, I have a collection of little scripts. This is an index to the ones I've actually written something about, mostly pointing to discussions in my techblog.

addr	Turn full hostnames into IP addresses (via DNS lookups).
countup	Feed field's output to howmany.
expsyslog.pl	Expand 'last message repeated N times' syslog messages.
field	Extract the given field(s) from stdin.
howmany	Sum up how many times lines occur on stdin.
nsaddrs	Report the IP addresses of the nameservers for a particular zone.
nsdig	Run dig against the nameservers for a particular zone.
sdig	dig with short output.
wcat	Dump a URL to standard out.

People on GPU can usually find these in /u/cks/bin or /u/cks/adm (depending). (/u/cks/bin has a hysterically out of date README for other things.)

Miscellaneous X tools and utilities I have accumulated

X has a crying lack of little utilities that you can use as the building blocks in shell scripts and so on. However, here's my little collection of bits and pieces, some of them original to me and some of them hoovered up from elsewhere.

tkmsg	Put up a window with a message; click on the window and it goes away. This is handy for notifications. (TCL/TK)
xmospos.c	Report the current X and Y positions of the mouse, so you can position new windows in the right spots. (C)

procage: when was a Linux process started

While ps will give you a summarized version of when a process started, it shows less and less precision the older the process is. It's sometimes useful to know precisely when this happened, especially for old processes; hence, procage.py.

Usage: procage.py [-a] PID [PID ...]

If given a -a argument, procage.py reports the process's age in seconds instead of its start time.

This is Linux specific, since it uses Linux's /proc information.

Patches I make to liferea

Liferea is my Unix feed reader of choice; it definitely runs on Linux and probably runs on anything with a recent enough set of GTK / GNOME libraries. However, I've found that I need to add some patches:

liferea-1.0.9-atom-title.patch	Fixes the bug where liferea truncates Atom 1.0 HTML entry titles that have HTML markup in them.
liferea-1.0.9-atom-times.patch	Makes liferea prefer the <published> time over the <updated> time for Atom 1.0 entries if both exist for an entry. Without this patch, liferea decides that the entry's time is whichever one was the last it saw. This patch also changes the English text for an element to be more applicable for both feeds and entries.

These patches are both against liferea 1.0.9, the current version as I write this, but they should apply with relatively little problems to prior 1.0.x versions. They should be applied in order, with patch -p1.

(At some point when I feel like fighting with SourceForge's bug tracker I will try to report both of these as bugs. Feel free to beat me to it.)

DWiki's source code

DWiki is my name for the wikioid thing that produces what you're reading right now.

Currently I don't have DWiki packaged for distribution. People with GPU accounts can see the live directory area for CSpace in /u/cks/web; the source code itself is /u/cks/web/prog. (Another copy of the source code, often with in development stuff, is in /u/cks/src/dwiki; I push to the production area only every so often.)

If you are interested in using DWiki, please get in touch with me. I'm happy to have it happen, but right now it needs a lot more documentation and a certain amount of handholding, and because it keeps evolving I'd like to keep track of who's got copies so I can tell you about new 'releases'.

(DWiki is written by Chris Siebenmann, with a significant contribution from Daniel Martin.)