Wandering Thoughts archives

2019-12-07: Some important things about how PCIe works out involve BIOS magic
2019-12-06: PCIe bus addresses, lspci, and working out your PCIe bus topology
2019-12-05: Looking into your system's PCIe slot topology and PCIe lane count under Linux
2019-12-04: Desktop motherboards can have fewer useful PCIe slots than they seem to
2019-12-02: You can have Grafana tables with multiple values for a single metric (with Prometheus)
Calculating usage over time in Prometheus (and Grafana)
2019-11-30: Operating spam and malware filtering is ultimately a social problem
Counting the number of distinct labels in a Prometheus metric
2019-11-29: The problem of multiple NVMe drives in a PC desktop today
2019-11-27: Selecting metrics to gather mostly based on what we can use
Capturing command output in a Bourne shell variable as a brute force option
2019-11-25: In Prometheus, don't be afraid of high cardinality metrics if they're valuable enough
2019-11-24: I use unit tests partly to verify that something works in the first place
Timing durations better in Python (most of the time)
2019-11-23: Thinking about timeouts and exceptions in Python
2019-11-22: Our problem of checking if our L2TP VPN servers are actually working
2019-11-20: In the old days, we didn't use multiple Unixes by choice (mostly)
The fading out of multi-'architecture' Unix environments
2019-11-18: LiveJournal and the path to NoSQL
2019-11-17: It's good to make sure you have notifications of things
The operational differences between notifications and logs
2019-11-15: How we structure our Django web application's configuration settings
2019-11-14: TCP/IP and a consequence of reliable delivery guarantees
2019-11-13: How to make a rather obnoxiously bad web spider the easy way
My mistake in forgetting how Apache .htaccess files are checked
2019-11-11: An apparent hazard of removing Linux software RAID mirror devices
2019-11-10: Putting a footer on automated email that says what generated it
The problems with piping curl to a shell are system management ones
2019-11-08: I have to assume that people here can be successfully phished
2019-11-07: Some notes on getting email when your systemd timer services fail
Realizing that Go constants are always materialized into values
2019-11-06: Systemd needs official documentation on best practices
2019-11-04: Systemd timer units have the unfortunate practical effect of hiding errors
Many of our 'worklog' messages currently assume a lot of context
2019-11-02: Using personal ruleset recipes in uMatrix in Firefox
2019-11-01: The appeal of text templating systems for generating HTML
2019-10-31: Some thoughts on the pragmatics of classifying phish spam as malware
2019-10-30: Chrony has been working well for us (on Linux, where we use it)
2019-10-29: Netplan's interface naming and issues with it
2019-10-28: One of XHTML's practical problems was its implications for web page generation
2019-10-27: An interesting little glitch in how Firefox sometimes handles updates to addons
My common patterns in shell script verbosity (for sysadmin programs)
2019-10-26: An incorrect superstition about running commands in the Bourne shell
2019-10-24: Third party ClamAV signatures seem to include a lot of phish and other spam
2019-10-23: The DBus daemon and out of memory conditions (and systemd)
2019-10-22: Groups of processes are a frequent and fundamental thing in Unix
2019-10-21: Filesystem size limits and the complication of when errors are detected
2019-10-20: A small irritation with Go's crypto/tls package
2019-10-19: Ubuntu LTS is (probably) still the best Linux for us and many people
2019-10-18: My little irritation with Firefox's current handling of 'Do-Not-Track'
Remembering that Django template code is not quite your Django Python code
2019-10-16: Some magical weirdness in Django's HTML form classes
2019-10-15: The Ubuntu package roulette
2019-10-14: Googlebot is both quite fast and very determined to crawl your pages
2019-10-13: If you don't test it in your automated testers, it's broken
Magic is fine if it's all magic: why I've switched to use-package in GNU Emacs
2019-10-11: A YAML syntax surprise and trick in Prometheus Alertmanager configuration
2019-10-10: Some additional information on ZFS performance as you approach quota limits
2019-10-09: Limiting the size of things in a filesystem is harder than it looks
2019-10-08: How we implement reboot notifications when our machines reboot in Prometheus
2019-10-07: Why we generate alert notifications about our machines having rebooted
2019-10-06: Automating our 'bookable' compute servers with SLURM has created generic 'cattle' machines
2019-10-05: The wikitext problem with new HTML elements such as <details>
2019-10-04: Vim, its defaults, and the problem this presents sysadmins
2019-10-03: Making changes to multiple files at once in Vim
2019-10-02: It's useful to record changes that you tried and failed to do
2019-10-01: My interest in and disappointment about HTML5's new <details> element
2019-09-30: Using alerts as tests that guard against future errors
Link: The asymmetry of Internet identity
ZFS performance really does degrade as you approach quota limits
2019-09-29: Understanding when to use and not use the -F option for flock(1)
2019-09-27: Some field notes on imposing memory resource limits on users on Ubuntu 18.04
A file permissions and general deployment annoyance with Certbot
2019-09-25: It's always convenient when malware is clear about its nature (7z edition)
Our workaround for Ubuntu 16.04 and 18.04 failing to reliably reboot some of our servers
2019-09-24: How we implement per-user CPU and memory resource limits on Ubuntu
2019-09-22: The increasing mess of libreadline versions in Linux distributions
2019-09-21: Why chroot is a security feature for (anonymous) FTP
2019-09-20: Modernizing (a bit) some of our HTML form <input> elements
TLS server certificate verification has two parts (and some consequences)
2019-09-18: Firefox, DNS over HTTPS, and us
Converting a Go pointer to an integer doesn't quite do what it looks like
2019-09-17: Finding metrics that are missing labels in Prometheus (for alert metrics)
2019-09-16: The problem of 'triangular' Network Address Translation
2019-09-14: Some notes on the structure of Go binaries (primarily for ELF)
2019-09-13: Bidirectional NAT and split horizon DNS in our networking setup
2019-09-12: The mystery of why my Fedora 30 office workstation was booting fine
2019-09-11: Making your own changes to things that use Go modules
2019-09-10: Catching Control-C and a gotcha with shell scripts
2019-09-09: A safety note about using (or having) go.mod inside $GOPATH in Go 1.13
2019-09-08: Jumping backward and forward in GNU Emacs
2019-09-07: CentOS 7 and Python 3
2019-09-06: Programs that let you jump around should copy web browser navigation
Go modules and the problem of noticing updates to dependencies
2019-09-04: If you use the rarfile module, make sure you're using version 3.0 (or later)
Using Wireshark's Statistics menu to get per-host traffic volume
2019-09-02: Another way to do easy configuration for lots of Prometheus Blackbox checks
2019-09-01: Some limitations of wifi MAC address randomization
2019-08-31: The sorts of email attachments that we get these days have become boring
2019-08-30: ZFS is not a universal filesystem that is always good for all workloads
How I'm dealing with my Python indentation problem in GNU Emacs
2019-08-28: A wifi MAC address randomization surprise in a new Android gadget
Allowing some Alias directives to override global Redirects in Apache
2019-08-26: A lesson of (alert) scale we learned from a power failure
2019-08-25: Text UIs and the problem of discoverability
2019-08-24: Apache, Let's Encrypt, and site-wide reverse proxies and HTTP redirections
2019-08-23: What happens in ZFS when you have 4K sector disks in an ashift=9 vdev
Link: GNOME Terminal Cursor Blinking Saga
2019-08-22: Pruning deleted remote Git branches (manually or automatically)
2019-08-21: Making sense of OpenBSD 'pfctl -ss' output for firewall state tables
2019-08-20: Saying goodbye to Flash (in Firefox, and in my web experience)
2019-08-19: Go 2 Generics: contracts are now boring (and that's good)
2019-08-18: Early notes on using LSP-based editing in GNU Emacs for Python
2019-08-17: A situation where Python has undefined values
2019-08-16: A gotcha with Fedora 30's switch of Grub to BootLoaderSpec based configuration
Systemd and waiting until network interfaces or addresses are configured
2019-08-15: Getting LSP-based editing working for Go in GNU Emacs
2019-08-13: Changes to Go and the appearance of finality
2019-08-12: Linux can run out of memory without triggering the Out-Of-Memory killer
2019-08-11: Roughly when the Linux Out-Of-Memory killer triggers (as of mid-2019)
2019-08-10: One core problem with DNSSEC
2019-08-09: Turning off DNSSEC in my Unbound instances
Non-uniform caches are harder to make work well
2019-08-07: What has to happen with Unix virtual memory when you have no swap space
Rewriting my iptables rules using ipsets
2019-08-05: dup(2) and shared file descriptors
2019-08-04: Some notes on understanding how to use flock(1)
2019-08-03: Sharing file descriptors with child processes is a clever Unix decision
Link: ASCII table and history (Or, why does Ctrl+i insert a Tab in my terminal?)
2019-08-02: If you can, you should use flock(1) for shell script locking
Getting NetworkManager to probably verify TLS certificates for 802.1x networks
2019-08-01: How not to set up your DNS (part 24)
2019-07-31: How mountd and exportfs handle NFS export permissions on Linux
Link: A program to read AMD Ryzen RAPL information on Linux
2019-07-30: I think it's time to explicitly set Go's $GO111MODULE environment variable
2019-07-29: The practical difference between CPU TDP and observed power draw illustrated
2019-07-28: A note on using the Go Prometheus client package to exposed labeled metrics
What I want out of my window manager
2019-07-26: Some things on the GUID checksum in ZFS pool uberblocks
How 'zpool import' generates its view of a pool's configuration
2019-07-24: I think I like systemd's DynamicUser feature (under the right circumstances)
ZFS pool imports happen in two stages of pool configuration processing
2019-07-22: Why file and directory operations are synchronous in NFS
2019-07-21: Why we're going to be using Certbot as our new Let's Encrypt client
Wireless networks have names and thus identify themselves
2019-07-19: Some brief views on iOS clients for Mastodon (as of mid 2019)
2019-07-18: Switching Let's Encrypt clients is currently quite disruptive
2019-07-17: Django 1.11 has a bug that causes intermittent CSRF validation failures
2019-07-16: Go's proposed try() will be used and that will change how code is written
2019-07-15: ZFS on Linux still has annoying issues with ARC size
2019-07-14: We're going to be separating our redundant resolving DNS servers
2019-07-13: Our switches can wind up in weird states after a power failure
2019-07-12: Browers can't feasibly stop web pages from talking to private (local) IP addresses
Reflections on almost entirely stopping using my (work) Yubikey
2019-07-10: I brought our Django app up using Python 3 and it mostly just worked
2019-07-09: Systemd services that always restart should probably set a restart delay too
2019-07-08: SMART drive self-tests seem potentially useful, but not too much
2019-07-07: Straightforward web applications are now very likely to be stable in browsers
2019-07-06: Clearing disk errors (or SMART complaints) for Linux software RAID arrays
2019-07-05: My plan for two-stage usage of Certbot when installing web server hosts
2019-07-04: Django's goals are probably not our goals for our web application
2019-07-03: Converting a variable to a single-element slice in Go via unsafe
Finding out what 32-bit x86 Linux programs your users are running
2019-07-01: The power of option types is in what they do to the rest of the language
2019-06-30: Understanding why 'root window' X under Wayland may matter
2019-06-29: Being realistic about what we're going to do with our Django app
2019-06-28: Using Prometheus's statsd exporter to let scripts make metrics updates
2019-06-27: Our last OmniOS fileserver is now out of production (and service)
2019-06-26: The death watch for the X Window System (aka X11) has probably started
A hazard of our old version of OmniOS: sometimes powering off doesn't
2019-06-24: The convenience (for me) of people writing commands in Python
2019-06-23: What it takes to run a 32-bit x86 program on a 64-bit x86 Linux system
2019-06-22: Google Groups entirely ignores SMTP time rejections
We get a certain amount of SMTP MAIL FROM's in UTF-8 with odd characters
2019-06-21: One of the things a metrics system does is handle state for you
2019-06-19: How Bash decides it's being invoked through sshd and sources your .bashrc
A Let's Encrypt client feature I always want for easy standard deployment
2019-06-17: Sometimes, the problem is in a system's BIOS
My Mastodon remark about tiling window managers
2019-06-16: Firefox and my views on the tradeoffs of using DNS over HTTPS
2019-06-15: Some notes on Intel's CPUID and how to get it for your CPUs
2019-06-14: Intel's approach to naming Xeon CPUs is pretty annoying
Intel's MDS issues have now made some old servers almost completely useless to us
2019-06-12: My weird problem with the Fedora 29 version of Firefox 67
An interesting Fedora 29 DNF update loop with the createrepo package
2019-06-10: Keeping your past checklists doesn't help unless you can find them again
2019-06-09: Go recognizes and specially compiles some but not all infinite loops
2019-06-08: Hardware Security Modules are just boxes running opaque and probably flawed software
Our current approach for updating things like build instructions
2019-06-06: Feed readers and their interpretation of the Atom 'title' element
2019-06-05: The HTML <pre> element doesn't do very much
Go channels work best for unidirectional communication, not things with replies
2019-06-03: Almost all of our OmniOS machines are now out of production
2019-06-02: Exploring the start time of Prometheus alerts via ALERTS_FOR_STATE
I haven't customized my Vim setup and I'm not sure I should try to (yet)
2019-05-31: Some things on how ZFS dnode object IDs are allocated (which is not sequentially)
Some things about where icons for modern X applications come from
2019-05-30: Conditional expressions in any form are an attractive thing
2019-05-28: Distribution packaging of software needs to be informed (and useful)
An interesting report on newly used domain names and their usage in spam
2019-05-27: Something that Linux distributions should not do when packaging things
2019-05-26: Why I no longer have strong feelings about changes to Python
2019-05-25: Blocking JavaScript by default is not an easy path
2019-05-24: The problem of paying too much attention to our dashboards
2019-05-23: I will probably never give my shell dotfiles the major reform they could use
On programming languages belonging (or not) to their community
2019-05-22: Go is Google's language, not ours
2019-05-20: Understanding how to pull in labels from other metrics in Prometheus
2019-05-19: DKIM signed email as a signal (of something)
2019-05-18: Binding keys to actions in xterm, and my bindings as an example
2019-05-17: My new favorite tool for looking at TLS things is certigo
One of our costs of using OmniOS was not having 10G networking
2019-05-16: Go has no type for types in the language
2019-05-15: An infrequent odd kernel panic on our Ubuntu 18.04 fileservers
2019-05-13: Fixing Alpine to work over NFS on Ubuntu 18.04 (and probably other modern Linuxes)
2019-05-12: What we'll want in a new Let's Encrypt client
Committed address space versus active anonymous pages in Linux: a mystery
2019-05-10: Some thoughts on Red Hat Enterprise 8 including Python 2 and what it means
Firefox versus Chrome (my 2019 view)
2019-05-08: Some general things and views on DNS over HTTPS
A Linux machine with a strict overcommit limit can still trigger the OOM killer
2019-05-07: Some weird and dubious syndication feed fetching from SBL-listed IPs
2019-05-05: TLS certificate rollover outside of the web is complex and tangled
What usually identifies an intermediate or root TLS certificate
2019-05-03: In Go, unsafe type conversions are still garbage collection safe
Some implications of using offset instead of delta() in Prometheus
2019-05-01: One of my problems with YAML is its sheer complexity
2019-04-30: My problem with YAML's use of whitespace
2019-04-29: Notifications and interruptions, and my view on them
2019-04-28: A gotcha with stale metrics and *_over_time() in Prometheus
2019-04-27: Some useful features of (GNU) date for things like time conversion
2019-04-26: Brief notes on making Prometheus instant queries with curl
Various aspects of Python made debugging my tarfile problem unusual
2019-04-24: How we're making updated versions of a file rapidly visible on our Linux NFS clients
The appeal of using plain HTML pages
2019-04-22: Go 2 Generics: The usefulness of requiring minimal contracts
You might as well get an x86 CPU now, despite Meltdown and its friends
2019-04-21: My view on upgrading Prometheus (and Grafana) on an ongoing basis
2019-04-19: V7 Unix programs are often not written the way you would expect
Links: A Practitioner's Guide to System Dashboard Design (with a bonus)
2019-04-18: One reason ed(1) was a good editor back in the days of V7 Unix
A pattern for dealing with missing metrics in Prometheus in simple cases
2019-04-17: Private browsing mode versus a browser set to keep nothing on exit
2019-04-15: How Linux starts non-system software RAID arrays during boot under systemd
2019-04-14: A VPN for me but not you: a surprise when tethering to my phone
2019-04-13: Remembering that Prometheus expressions act as filters
WireGuard was pleasantly easy to get working behind a NAT (or several)
2019-04-12: Getting (and capturing) spam can sometimes be useful to see what's in it
2019-04-10: The tarfile module is too generous about what is considered a tar file
A Git tool that I'd like and how I probably use Git differently from most people
2019-04-08: An example of a situation where Go interfaces can't substitute for generics
2019-04-07: Why selecting times is still useful even for dashboards that are about right now
A ZFS resilver can be almost as good as a scrub, but not quite
2019-04-05: I won't be trying out ZFS's new TRIM support for a while
It's always DNS (a story of our circular dependency)
2019-04-03: A sign of people's fading belief in RSS syndication
NVMe and an interesting technology change
2019-04-01: Our plan for handling TRIM'ing our ZFS fileserver SSDs
2019-03-31: Our likely ZFS fileserver upgrade plans (as of March 2019)
Erasing SSDs with blkdiscard (on Linux)
2019-03-29: Our current approach for significantly upgrading or modifying servers
My NVMe versus SSD uncertainty (and hesitation)
2019-03-27: A new and exciting failure mode for Linux UEFI booting
2019-03-26: Drifting away from OmniOS (CE)
2019-03-25: The mystery of my desktop that locks up when it gets too cold
2019-03-24: Prometheus's delta() function can be inferior to subtraction with offset
Link: What has your microcode done for you lately?
A bit more on ZFS's per-pool performance statistics
2019-03-22: Sometimes the simplest version of a graph is a text table
2019-03-21: What sorts of good email attachments our users get (March 2019 edition)
2019-03-20: The types of attachments we see in malware email (March 2019 edition)
2019-03-19: ZFS Encryption is still under development (as of March 2019)
2019-03-18: Prometheus subqueries pick time points in a surprising way
2019-03-17: Going from a bound instance method to its class instance in Python
2019-03-16: Callable class instances versus closures in Python
Staying away from Google Chrome after six months or so
2019-03-14: The plague of 'you've logged in to our site again' notification emails
2019-03-13: Peculiarities about Unix's statfs() or statvfs() API
2019-03-12: An easy optimization for restricted multi-metric queries in Prometheus
2019-03-11: Testing Prometheus alert conditions through subqueries
2019-03-10: What the default query step is for Prometheus subqueries
Turning something into a script encourages improving it
2019-03-08: Exploring how and why interior pointers in Go keep entire objects alive
2019-03-07: Exploring the mild oddity that Unix pipes are buffered
Our problem with Netplan and routes on Ubuntu 18.04
2019-03-06: Using Prometheus subqueries to look for spikes in rates
2019-03-04: A surprisingly arcane little Unix shell pipeline example
2019-03-03: Understanding a change often requires understanding how the code behaves
Really understanding diffs requires knowing their context too
2019-03-01: What you get when you do a DNS A record lookup for a CNAME'd name
2019-02-28: Taking advantage of the Linux kernel NFS server's group membership cache
2019-02-27: How to see and flush the Linux kernel NFS server's group membership cache
Using Prometheus subqueries to do calculations over time ranges
2019-02-25: ntpdate has a surprising restriction on what it will sync to
Link: Vim anti-patterns
2019-02-24: Process states from /proc/[pid]/stat versus /proc/stat's running and blocked numbers
The modern danger of locales when you combine sort and cron
2019-02-22: Using default function arguments to avoid creating a class
2019-02-21: An advantage of tablets and two-in-one devices over small laptops
2019-02-20: What /proc/[pid]/stat's process state means and where it comes from
The cliffs in the way of adding tests to our Django web app
2019-02-18: When cloning git repos, things go faster if you start from a good base
2019-02-17: Why I like middle mouse button paste in xterm so much
Some notes on heatmaps and histograms in Prometheus and Grafana
2019-02-15: Accumulating a separated list in the Bourne shell
2019-02-14: A pleasant surprise with a Thunderbolt 3 10G-T Ethernet adapter
2019-02-13: An unpleasant surprise with part of Apache's AllowOverride directive
2019-02-12: Using grep with /dev/null, an old Unix trick
2019-02-11: Thinking about the merits of 'universal' URL structures
2019-02-10: Open protocols can evolve fast if they're willing to break other people
2019-02-09: 'Scanned' versus 'issued' numbers for ZFS scrubs (and resilvers)
2019-02-08: Making more use of keyboard control over window position and size
2019-02-07: A touchpad is not a mouse, or at least not a good one
2019-02-06: Using a single git repo to compare things between two upstreams
2019-02-05: A problem with strict memory overcommit in practice
2019-02-04: Hand-building an updated upstream kernel module for your (Fedora) kernel
2019-02-03: My temptation of getting a personal laptop
2019-02-02: A little appreciation for Vim's 'g' command
2019-02-01: A bit of Sun's history that still lingers on in Illumos
2019-01-31: What getopt package I use for option handling in my Go programs
2019-01-30: ZFS On Linux's kernel modules issues are not like NVidia's
How having a metrics system centralized information and got me to check it
2019-01-28: Go 2 Generics: some features of contracts that I like
The potential risk to ZFS created by the shift in its userbase
2019-01-27: How to handle Unicode character decoding errors depends on your goals
2019-01-25: A piece of email malware that wanted to make sure we rejected it
The Linux kernel's pstore error log capturing system, and ACPI ERST
2019-01-23: Consider setting your Linux servers to reboot on kernel problems
A little surprise with Prometheus scrape intervals, timeouts, and alerts
2019-01-22: Things you can do to make your Linux servers reboot on kernel problems
2019-01-21: Two annoyances I have with Python's imaplib module
2019-01-20: A few notes on using SSL in Python 3 client programs
2019-01-19: A surprise potential gotcha with sharenfs in ZFS on Linux
2019-01-18: Linux CPU numbers are not necessarily contiguous
2019-01-17: Why C uninitialized global variables have an initial value of zero
2019-01-16: Perhaps you no longer want to force a server-preferred TLS cipher order on clients
2019-01-14: Why your fresh new memory pages are zero-filled
2019-01-13: Two views of ZFS's GPL-incompatibility and the Linux kernel
The risk that comes from ZFS on Linux not being GPL-compatible
Even thinking about spam makes me angry
2019-01-12: I have somewhat mixed feelings about Python 3's socket module errors
2019-01-11: A new drawback of using my custom-compiled Firefox
2019-01-10: Why I still have a custom-compiled Firefox (early 2019 edition)
2019-01-09: On right and wrong ways to harvest system-level performance stats
2019-01-08: Link: The IOCCC 2018 "Best of show" program
2019-01-07: Daemons and the pragmatics of unexpected error values from system calls
2019-01-06: accept(2)'s problem of trying to return two different sorts of errors
Linux network-scripts being deprecated is a problem for my home PPPoE link
2019-01-05: Some things on ZFS's per-pool performance statistics
2019-01-04: Planning ahead in documentation worked out for us
2019-01-02: You shouldn't allow Firefox to recommend things to you any more
How I get a copy of the Ubuntu kernel source code (as of Ubuntu 18.04)

Page tools: See As Normal.
Search:
Login: Password:
Atom Syndication: Recent Pages, Recent Comments.

This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.