Wandering Thoughts archives


Why I like middle mouse button paste in xterm so much

In my entry about how touchpads are not mice, I mused that one of the things I should do on my laptop was insure that I had a keyboard binding for paste, since middle mouse button is one of the harder multi-finger gestures to land on a touchpad. Kurt Mosiejczuk recently left a comment there where they said:

Shift-Insert is a keyboard equivalent for paste that is in default xterm (at least OpenBSD xterm, and putty on Windows too). I use that most of the time now as it seems less... trigger-happy than right click paste.

This sparked some thoughts, because I can't imagine giving up middle mouse paste if I have a real choice. I had earlier seen shift-insert mentioned in other commentary on my entry and so have tried a bit to use it on my laptop, and it hasn't really felt great even there; on my desktops, it's even less appealing (I tried shift-insert out there to confirm that it did work in my set of wacky X resources).

In thinking about why this is, I came to the obvious realization about why all of this is so. I like middle mouse button paste in normal usage because it's so convenient, because almost all of the time my hand is already on the mouse. And the reason my hand is already on the mouse is because I've just used the mouse to shift focus to the window I want to paste into. Even on my laptop, my right hand is usually away from the keyboard as I move the mouse pointer on the touchpad, making shift-Insert at least somewhat awkward.

(The exception that proves the rule for me is dmenu. Dmenu is completely keyboard driven and when I bring it up, Ctrl-Y to paste the current X selection is completely natural.)

I expect that people who use the keyboard to change window focus have a pretty different experience here, whether they're using a fully keyboard driven window manager or simply one where they use Alt-Tab (or the equivalent) to select through windows. My laptop's Cinnamon setup has support for Alt-Tab window switching, so perhaps I should try to use it more. On the other hand, making the text selection I'm copying is generally going to involve the mouse or touchpad, even on my laptop.

(I don't think I want to try keyboard window switching in my desktop fvwm setup for various reasons, including that I think you want to be using some version of 'click to focus' instead of mouse pointer based focus for this to really work out. Having the mouse pointer in the 'wrong' window for your focus policy seems like a recipe for future problems and unpleasant surprises. On top of that, X's handling of scroll wheels means that I often want the mouse pointer to be in the active window just so I can use my mouse's scroll wheel.)

PS: Even if it's possible to use keyboard commands to try to select things in xterm or other terminal emulators, I suspect that I don't want to bother trying it. I rather expect it would feel a lot like moving around and marking things in vi(m), with the added bonus of having to remember an entire different set of keystrokes that wouldn't work in Firefox and other non-terminal contexts.

unix/MouseMovementAndPaste written at 22:46:50; Add Comment

Some notes on heatmaps and histograms in Prometheus and Grafana

On Mastodon (or if you prefer, the Fediverse), I mentioned:

I have now reached the logical end point of running Prometheus on my desktop, which is that I have installed Grafana so I can see heatmap graphs of my disk IO latency distributions generated from the Cloudflare eBPF exporter.

It's kind of neat once I got all the bits going.

This isn't my first go-around on heatmaps and histograms, but this time around I found new clever mistakes to make on top of my existing confusions. So it's time for some notes, in the hopes that they will make next time easier.

Grafana can make heatmaps out of at least two different sorts of Prometheus metrics, showing the distribution of numeric values over time (a value heatmap). The first sort, which is simpler and the default if you set up a heatmap panel, is gauges or gauge-like things, such as the number of currently active Apache processes or the amount of CPU usage over the past minute (which you would generate with rate() from the underlying counters). You could visualize these metrics in a conventional graph, but in many cases the graph would wiggle around madly and it would be hard to see much in it. Showing the same data in a heatmap may provide more useful and readable information.

When used this way, Grafana automatically works out the heatmap buckets to use from the data values and groups everything together and it is all very magical. Grafana takes multiple samples for every bucket's time range, but not all that many samples, and there is no real way to control this. In particular, as the time range goes up Grafana will sample your metric at steadily courser resolution, even though it could use a finer resolution to get more detailed information for buckets. As a consequence, for gauges you almost certainly want to use avg_over_time or max_over_time instead of the raw metric.

(Using rate() on a counter already gives you this implicitly.)

The other sort of Grafana heatmaps are made from Prometheus histogram metrics, which the Grafana documentation calls 'pre-bucketed data'. With these, you have to go to the Axes tab of the panel and set the "Data format" to "Time series buckets", and you also normally set the "Legend format" to '{{le}}' in the Metrics tab so that the Y axis can come out right. Failing to change the data format will give you very puzzling heatmaps and it is not at all obvious what's wrong and how you fix it.

(It's a real pity that Grafana doesn't auto-detect that this is a Prometheus histogram metric and automatically switch the data format and so on for you. It would make things much more usable and friendly.)

Prometheus histogram metrics can be either counters or gauges. A histogram of the number of IMAP connections per user would be a gauge histogram, because it changes up and down as people log off. A histogram of disk IO latency is a counter histogram; it will normally only count up. You need to rate() or increase() counter histograms in order to get useful heatmap displays; gauge histograms can be used as-is, although you probably want to consider running them through avg_over_time or max_over_time.

(Prometheus's metric type information doesn't distinguish between these two sorts of histograms. If you're lucky, the HELP text for a particular histogram tells you which it is; if you're not, you get to deduce it from what the histogram is measuring and how it behaves over time.)

One easy to make mistake is to have your heatmap metric query in Grafana actually return more than one metric sequence. For instance, when I first set up a heatmap for my disk latency metrics, I didn't realized that they came in a 'read' and a 'write' version for each disk. The resulting combined heatmap was rather confusing, with all sorts of nonsensical bucket counts. In theory you can put such multiple metrics in the same heatmap by creating separate names in the legend format, for example '{{le}} {{operation}}', but in practice this gives you two (or more) heatmaps stacked on top of each other and is not necessarily what you want. As far as I know, there's no way to combine two metrics or superimpose two metrics in the same heatmap. Sadly, this does result in an explosion of heatmaps for things like disk latency, so you probably want to use some Grafana dashboard variables to select what disk (or perhaps disks) you want heatmaps from.

It seems surprisingly hard to find a colour scheme for Grafana heatmaps that both has a pleasant variation from common to uncommon values while still clearly showing that uncommon values are present. By default, Grafana seems to want to fade uncommon values out almost to invisibility, which is not what I want; I want uncommon values to stand out, because they are one of the important things I'm looking for with heatmaps and histograms in general. Perhaps this is a sign that Grafana heatmaps are not actually the best way of looking for unusual values in Prometheus histograms, although they are probably a good way of looking at details once I know that some are present.

(I've also learned some hard lessons about hand-building histogram metrics for Prometheus. My overall advice there is to delegate the job to someone else's code if you have the choice, because it's really hard to get right if you're doing it all yourself.)

PS: for things like disk IO latency distributions, where the tail end is multiple seconds but involves fractions like '2.097152', it helps to explicitly set the Y axis decimals to '1' instead of leaving it on auto. This helps the Y axis label take up less space so the buckets get more of it. For disk IO sizes, I even set the decimals to '0'. Grafana's obsession with extreme precision in this sort of thing is both impressive and irksome.

sysadmin/PrometheusGrafanaHeatmaps written at 00:47:08; Add Comment

Page tools: See As Normal.
Login: Password:
Atom Syndication: Recent Pages, Recent Comments.

This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.