Chris's Wiki :: blog Commentshttps://utcc.utoronto.ca/~cks/space/blog/?atomcommentsDWiki2024-03-18T20:39:25ZRecent comments in Chris's Wiki :: blog.By Chris Siebenmann on /blog/tech/WriteBufferingAndSyncstag:CSpace:blog/tech/WriteBufferingAndSyncs:091bcc2cc9dafc51c230846b96b4fe66df01e629Chris Siebenmann<div class="wikitext"><p>I don't think there's any fundamental obstacle to a filesystem making
it so that committing the journal isn't a choke point. But at the same
time I don't think very many do it, and I think it's probably easier to
implement it as basically a single-threaded process. If you implement
journal commit as a concurrent process you need to carefully keep various
things separate even if they'd normally be mingled together (for example,
allocating new space for new data blocks).</p>
</div>2024-03-18T20:39:25ZBy George Spelvin on /blog/tech/WriteBufferingAndSyncstag:CSpace:blog/tech/WriteBufferingAndSyncs:39201b9fe5db26e723550dca0054d7ab7d74ae80George Spelvin<div class="wikitext"><p>It seems to me that there's a simpler solution: When <code>fsync()</code>ing, empty the buffers <em>before</em> synchronizing with other journal writes.</p>
<p>Just to state explicitly what's implicit in what you wrote, there's a difference between the data being on disk, and the associated metadata being written. The second part is the "commit" which makes the write durable.</p>
<p>All file systems have this distinction, but journaling file systems make commits global, so you have more interference between writers.</p>
<p>Writing n blocks of data takes O(n) time, while the metadata commit is, if not quite O(log n), at least o(n). Large commits aren't themselves a prospect to be feared.</p>
<p>Keeping an overhang in RAM is useful <em>if</em> we have enough buffer space to absorb the write <em>and</em> we won't be synchronizing the write so can move on while the OS completes the writes asynchronously.</p>
<p>Given modern RAM sizes, the former threshold is quite generous, but we still need heuristics. It's annoying when one massive writer eats all the available RAM, stalling a lot of other smaller writers which could otherwise have proceeded asynchronously.</p>
<p>But I don't see why we need to make heuristic guesses at the second.</p>
<p>Rather, divide <code>fsync()</code> operations into two phases:</p>
<ol><li>Writing out the data</li>
<li>Committing the metadata</li>
</ol>
<p>The important part of this idea is that <em>phase 1 does not block journal commits.</em> Multiple other writers may force a journal commit while this lengthy preliminary is in progress. Only once it's on disk do we need to proceed to the associated global journal commit, which requires synchronization with other writers, but is never huge.</p>
<p>Rather than the awkward heuristic of saying "I suspect this process will want to sync its writes, so let's minimize RAM buffering", you wait until you have an <code>fsync()</code> call which tells you unambiguously. But then you flush the buffers <em>without blocking other syncs</em>, just like you would have done had your heuristic triggered on the initial <code>write()</code> call, until the final o(n) metadata update.</p>
</div>2024-03-18T14:31:56ZBy Chris Siebenmann on /blog/linux/SystemdStallAfterTooFastRestartstag:CSpace:blog/linux/SystemdStallAfterTooFastRestarts:8195cf682243a6fb16ce4a3950bd164c071b9d1bChris Siebenmann<div class="wikitext"><p>We wound up sticking a 'sleep' in the script because it was the easiest
way. The script takes multiple arguments already, so having it take
multiple multiple arguments is a stretch and the complexity of getting
all of that right would possibly make people tilt toward the 'run it
several times' approach since that's simpler to inspect and automate.</p>
</div>2024-03-17T19:38:58ZBy Aristotle Pagaltzis on /blog/linux/SystemdStallAfterTooFastRestartstag:CSpace:blog/linux/SystemdStallAfterTooFastRestarts:7d2a351900b86c5dd75bb6e200a1aa59ec13b8adAristotle Pagaltzishttp://plasmasturm.org/<div class="wikitext"><p>So which option did you adopt?</p>
</div>2024-03-17T07:23:18ZBy Mike Kohne on /blog/sysadmin/PrometheusDNSMonitoringProblemtag:CSpace:blog/sysadmin/PrometheusDNSMonitoringProblem:e692c030d7bbc0d5a7b201f365ad5758085db256Mike Kohne<div class="wikitext"><p>Have you considered a script to generate the various configs? It'd be easier than writing a whole exporter, yet probably a lot quicker.</p>
</div>2024-03-16T14:15:13ZBy Verisimilitude on /blog/programming/RustIsInevitabletag:CSpace:blog/programming/RustIsInevitable:6d7af8b05350b2f9ec9b327e2dfd6bf9ae779c35Verisimilitudehttp://verisimilitudes.net<div class="wikitext"><blockquote><p>In that way it's just as harmful and also just as inevitable as the migration from HTTP to HTTPS for websites.</p>
</blockquote>
<p>Another similarity is the total ignorance of better solutions to these problems. The TLS nonsense obscures cryptographic signing that works, and the Rust nonsense obscures the existence of Ada that worked better decades ago.</p>
<p>I'm thoroughly convinced the reason Rust is pushed so fervently is to make it impossible to build things from source code easily, not that it was easy beforehand.</p>
</div>2024-03-15T21:09:11ZBy Verisimilitude on /blog/web/HTTPSEverywherePragmaticstag:CSpace:blog/web/HTTPSEverywherePragmatics:ed7ef0093ef57ecce338d2bbe668a74286cc9f61Verisimilitudehttp://verisimilitudes.net<div class="wikitext"><p>Obeying the will of corporations which pushed this TLS nonsense to protect advertisements isn't pragmatism.</p>
<p>Roy T. Fielding really took the words out of my fingertips. All of this is done for the benefit of corporations, and for no other reason.</p>
</div>2024-03-15T20:16:48ZBy Verisimilitude on /blog/tech/PerfectionTraptag:CSpace:blog/tech/PerfectionTrap:e73489e34246557b4807e49863140a7386459690Verisimilitudehttp://verisimilitudes.net<div class="wikitext"><p>Notice no one ever says that the perfect mathematical proof is the enemy of the good mathematical proof. This is because a flawed mathematical proof is worthless, and similarly for software.</p>
<blockquote><p>In reality it's not actually a choice between right and worse; it's really a choice between nothing, worse, and right.</p>
</blockquote>
<p>In the eighteen years since and a world increasingly reliant on software with no fallback, it should be clear that nothing is preferable to deeply-flawed software.</p>
<blockquote><p>The easiest place to see this is computer security, where insistence on perfection (or some excellent approximation) is one of the holy tenets.</p>
</blockquote>
<p>It's better to know one is <em>insecure</em>, than to falsely believe one to be <em>secure</em>, however these words are defined.</p>
</div>2024-03-15T19:09:25ZBy rdiaz02 on /blog/programming/EmacsUnderstandingCompletiontag:CSpace:blog/programming/EmacsUnderstandingCompletion:c5a6843023fe2e2f80fc5e2145be300aacde2e4crdiaz02<div class="wikitext"><p>Thanks for the post. Two quick comments:</p>
<p>- "To use company completion, the buffer must be in company-mode, but you can disable as you type autocomplete with a buffer local value for 'company-begin-commands'.)" It might be easier to just "(setq company-idle-delay nil)".</p>
<p>- 'as you type' autocompletion : this seems possible with corfu too by setting the "corfu-auto" variable</p>
</div>2024-03-15T15:41:28ZBy Michael Warkentin on /blog/linux/SystemdNetworkdResetsIpRulestag:CSpace:blog/linux/SystemdNetworkdResetsIpRules:5125143aea4c17655ed1410a21600931d39bb538Michael Warkentin<div class="wikitext"><p>This was actually the cause of the global Datadog outage last year!</p>
<p><a href="https://www.datadoghq.com/blog/engineering/2023-03-08-deep-dive-into-incident-response/">https://www.datadoghq.com/blog/engineering/2023-03-08-deep-dive-into-incident-response/</a>
<a href="https://www.datadoghq.com/blog/2023-03-08-multiregion-infrastructure-connectivity-issue/#root-cause">https://www.datadoghq.com/blog/2023-03-08-multiregion-infrastructure-connectivity-issue/#root-cause</a></p>
</div>2024-03-14T14:46:41ZFrom 193.219.181.219 on /blog/linux/SystemdNetworkdResetsIpRulestag:CSpace:blog/linux/SystemdNetworkdResetsIpRules:b2640e5ff9319c9fadde7711ebd479545dc993edFrom 193.219.181.219<div class="wikitext"><blockquote><p>This is a reasonably sensible decision, both to deal with changes from previously specified routing policies and to also give people a way to clean out their experiments and reset to a known good base state. Similar logic applies to routes.</p>
</blockquote>
<p>In part it's probably also done because rules did not have (until very recently) a <code>proto</code> field that would allow networkd to distinguish "rules it has added before restart" from "rules added externally" (like how routing daemons generally flush only their own routes but not foreign routes).</p>
<p>Of course, having that field clearly doesn't stop networkd from flushing everything regardless, but it's still a prerequisite if it was ever made to not do so by default.</p>
<blockquote><p>and on top of that Netplan itself has limitations on what routing policies you can express (pushing you even more towards running 'ip rule' yourself).</p>
</blockquote>
<p>I assume Ubuntu still lets you ignore Netplan and write .network files by hand ? (Either that, or write <code>foo.network.d/</code> drop-ins to extend the Netplan-generated configs, if it comes to that...)</p>
</div>2024-03-14T06:49:36ZFrom 100.6.91.96 on /blog/linux/PrometheusHostAgentNonRootLossestag:CSpace:blog/linux/PrometheusHostAgentNonRootLosses:d292bb57541db77bc95761117ec572c4df03a2d5From 100.6.91.96<div class="wikitext"><p>It would seem that for node_dmi_info, one also loses some DMI info:</p>
<pre>
# ls -l /sys/class/dmi/id/
total 0
-r--r--r-- 1 root root 4096 Jan 11 06:01 bios_date
-r--r--r-- 1 root root 4096 Jan 12 22:05 bios_release
-r--r--r-- 1 root root 4096 Jan 11 06:01 bios_vendor
-r--r--r-- 1 root root 4096 Jan 11 06:01 bios_version
-r--r--r-- 1 root root 4096 Jan 11 06:01 board_name
-r-------- 1 root root 4096 Jan 11 06:01 board_serial
-r--r--r-- 1 root root 4096 Jan 11 06:01 board_vendor
-r--r--r-- 1 root root 4096 Jan 12 22:05 board_version
-r--r--r-- 1 root root 4096 Jan 11 06:01 chassis_asset_tag
-r-------- 1 root root 4096 Jan 12 22:05 chassis_serial
-r--r--r-- 1 root root 4096 Jan 11 06:01 chassis_type
-r--r--r-- 1 root root 4096 Jan 12 22:05 chassis_vendor
-r--r--r-- 1 root root 4096 Jan 12 22:05 chassis_version
-r--r--r-- 1 root root 4096 Mar 13 20:55 modalias
drwxr-xr-x 2 root root 0 Mar 13 20:55 power
-r--r--r-- 1 root root 4096 Jan 12 22:05 product_family
-r--r--r-- 1 root root 4096 Jan 2 13:11 product_name
-r-------- 1 root root 4096 Jan 11 06:01 product_serial
-r--r--r-- 1 root root 4096 Jan 12 22:05 product_sku
-r-------- 1 root root 4096 Jan 11 06:01 product_uuid
-r--r--r-- 1 root root 4096 Jan 12 21:57 product_version
lrwxrwxrwx 1 root root 0 Mar 13 20:55 subsystem -> ../../../../class/dmi
-r--r--r-- 1 root root 4096 Jan 2 13:06 sys_vendor
-rw-r--r-- 1 root root 4096 Mar 13 20:55 uevent
</pre>
<p>Why the kernel does this I can't imagine. I fear I'm going to have to have a boot-time cron job chmod u+r</p>
</div>2024-03-13T21:07:45ZFrom 193.219.181.219 on /blog/web/TLSCertsWhatIsManualtag:CSpace:blog/web/TLSCertsWhatIsManual:094cbbf944b7ed0f653211b7953a315404ad423cFrom 193.219.181.219<div class="wikitext"><blockquote><p>I believe that at least some TLS Certificate Authorities support automated issuance of year long certificates, but I'm not sure. Now that I've looked, I'm going to have to stop assuming that a website using a year-long TLS certificate is a reliable sign that they're not using automated issuance.</p>
</blockquote>
<p>Sectigo does; we're using it through GÉANT's account for universities. They bind ACME (and I believe also SCEP) to the regular account so you set up domains beforehand and there's no HTTP-challenge or anything during issuance; handy for getting certificates on behalf of some device that won't speak ACME on its own. (Yes, I know DNS challenge exists but still.)</p>
<p>The previous provider, DigiCert, was also halfway through setting up their ACME support (at that time for 3-year certs still) before the switchover; they also had their custom API as well, we never got around to writing tools for that, but I know at least one other university was using the API for some issuances.</p>
</div>2024-03-13T05:01:25ZBy Alex on /blog/programming/ShellPipelineStepsAndCPUstag:CSpace:blog/programming/ShellPipelineStepsAndCPUs:18d32f8d427d1e7bdb6269dc0c81bcbea4577025Alex<div class="wikitext"><blockquote><p>possibly un-compressing the files in place in parallel will work</p>
</blockquote>
<p>I don't know if you've noticed ripgrep's -z flag, but if not, it supports gz/bz2/xz/etc and it'll decompress multiple files in parallel. It's great if a grep is what you need.</p>
</div>2024-03-10T23:55:23ZBy Walex on /blog/sysadmin/MyDesktopTourtag:CSpace:blog/sysadmin/MyDesktopTour:5f11a020ff12a61d365adf4fb0f1e36d83f7509fWalex<div class="wikitext"><p>«<em>Keeping a file for this long is something I hope to achieve but finding it year later is the real problem. I Guess I'll research a good folder structure.</em>»</p>
<p>I have files going back to 1982 in my home directory and to 1976 on paper.
I use both year-based directories and topic-based ones, and not too many.
I also use a very good (but to do that it takes 10-15% of extra space) local spider/indexer called "recoll" which I have found works a lot better than most others:</p>
<p><a href="http://recoll.org/pages/index-recoll.html">http://recoll.org/pages/index-recoll.html</a></p>
</div>2024-03-10T21:39:37ZBy TheSameWayTheBricksDont on /blog/sysadmin/MyDesktopTourtag:CSpace:blog/sysadmin/MyDesktopTour:09d0f574cbd95ff23f2548f681a75663bbe4ab01TheSameWayTheBricksDont<div class="wikitext"><p>Incredible, thank you very much.
Keeping a file for this long is something I hope to achieve but finding it year later is the real problem.
I Guess I'll research a good folder structure.</p>
</div>2024-03-10T15:06:57ZBy Pete on /blog/linux/AMDWithECCKernelMessagestag:CSpace:blog/linux/AMDWithECCKernelMessages:46c8bb0ce4a26c46acbab6a2160ad2e5166dba89Pete<div class="wikitext"><p>A recent AMD EPYC platform has them under </p>
<p><code>/sys/devices/system/edac/mc/mc?/rank??</code></p>
</div>2024-03-08T21:29:32ZBy Chris Siebenmann on /blog/sysadmin/MyDesktopTourtag:CSpace:blog/sysadmin/MyDesktopTour:c4cc96f7a32561297d92e23992c1c1813a971607Chris Siebenmann<div class="wikitext"><p>I'm a data packrat so I have indeed kept the image. Doing some searches,
it appears to be a piece of concept art for Kiki by Katsuya Kondō. You
can see versions of it eg <a href="https://ghibli.fandom.com/wiki/Kiki">here</a>,
<a href="https://www.zerochan.net/3720670">here</a>, and <a href="https://ftp.sunet.se/mirror/archive/ftp.sunet.se/pub/tv+movies/anime-manga/new/">here</a>,
the last of which which may be my original source (no warranties implied
by any link). I suspect all of these versions were scanned from an art
book of the movie at some point many years ago and have circulated on
the Internet ever since.</p>
</div>2024-03-08T19:18:06ZBy TheSameWayTheBricksDont on /blog/sysadmin/MyDesktopTourtag:CSpace:blog/sysadmin/MyDesktopTour:d6d9530b9056976e878de9fa65dc26a6441d03b7TheSameWayTheBricksDont<div class="wikitext"><p>I know this is a (very) long shot but do you happen to have kept the Kiki wallpaper from your old desktop writeup? It was linked to on your old site but the FTP server serving the file doesn't exist anymore.
I love the movie and the drawing seem fantastic.</p>
<p>I just discovered you blog yesterday and I love it by the way, been looking around and following links ever since.</p>
</div>2024-03-08T17:41:07ZBy Adam D. Ruppe on /blog/unix/XWindowsAllTheWayDowntag:CSpace:blog/unix/XWindowsAllTheWayDown:e04f8069b33b246c0e925eb764fda574fc252af9Adam D. Ruppe<div class="wikitext"><p>"(I think toolkits may create some invisible sub-windows for event handling reasons. Gnome-terminal and other Gnome applications appear to create a 1x1 sub-window, for example.)"</p>
<p>Indeed. The reason for this is an interesting application and it took me quite a bit of digging to find it when I was faced with the problem it solves.</p>
<p>When you have child windows, keyboard events are sent to the first window, working from the descendant child up to the parent window, that is both under the current mouse pointer and subscribed to key events (aka "focus follows mouse") unless an explicit focus has been set to another sibling.</p>
<p>Suppose you want to embed another application in your window (something I still think is underutilized!). This other application will subscribe to key events, so unless you set an explicit focus to a valid sibling, it is going to get those events on a focus-follows-mouse model. This might be obnoxious for the user - move the mouse and the key events now don't go where you want - and it might be annoying for your application, since you no longer get the events on your top window. meaning things like your menu keyboard shortcuts stop working.</p>
<p>But there's a solution: that latter "unless". If the top-level window creates this extra window outside the rest of the child tree. It is still a child application's top-level parent, but not a child of anything else in there... making it a valid recipient of all these events with an explicit focus call.</p>
<p>Your application sets the focus to this child any time the top level thing gets it, and processes events through it. You then dispatch as you want - either sending it to the child widget handlers internally, or XSendEvent it to child windows as you wish, and things just work.</p>
<p>(Note that even if you don't embed other applications and don't use child windows as widgets, you probably want child windows for things like popup menus, so the technique is generally useful.)</p>
<p>I wrote more details on the stack overflow a couple years ago here: <a href="https://stackoverflow.com/questions/71544036/can-i-change-the-focus-behavior-of-child-windows/71800780#71800780">https://stackoverflow.com/questions/71544036/can-i-change-the-focus-behavior-of-child-windows/71800780#71800780</a> or you can go to the XEmbed spec on freedesktop directly which is the primary source I cite in there.</p>
</div>2024-03-07T23:31:20ZBy Chris Siebenmann on /blog/unix/XWindowsAllTheWayDowntag:CSpace:blog/unix/XWindowsAllTheWayDown:313bdeb0d590784e2dd816a4aae88e7af3744ba2Chris Siebenmann<div class="wikitext"><p>I think you're talking about a different thing, what I'll call 'window
manager windows'. Programs can use one visible window with UI controls
inside it, or separate windows, some with UI controls and some with
content (which could let you make the content full-size on one display
and park the controls on another). They can also use a <a href="https://en.wikipedia.org/wiki/Multiple-document_interface">MDI style UI</a>, where
they render visible sub-windows inside their main window and let
you move and resize those sub-windows.</p>
<p>(Although it's not MDI, modern web applications sometimes have similar
MDI-like window style objects that you can grab and move around inside
the web page. These are obviously not window manager windows, they're
entirely rendered in HTML and manipulated through DOM and Javascript.)</p>
<p>However, in X this is almost completely separate from how many
protocol-level Window objects the program is using. Every separate top
level 'window manager window' has to be a separate protocol-level
Window object, but programs can use or not use further Window
objects inside their top level windows as they want. This is almost
completely independent from how the UI looks, and you can't tell how
many protocol-level Window objects an X program is using from looking
at its UI; you have to look at protocol level things with tools like
'<code>xwininfo</code>'.</p>
</div>2024-03-06T16:10:57ZBy Anonymous on /blog/unix/XWindowsAllTheWayDowntag:CSpace:blog/unix/XWindowsAllTheWayDown:3709e48f3bc51fd2c57596f5bfab284f45412b05Anonymous<div class="wikitext"><p>'The Gimp' used to do this (even on MS-Windows), and even though these days the default is a single window, there still is an option to get the multiple windows GUI back.
<a href="https://www.gimp.org/">https://www.gimp.org/</a></p>
</div>2024-03-06T13:41:25ZBy B.Preston on /blog/unix/XServerBackingStoreOptionaltag:CSpace:blog/unix/XServerBackingStoreOptional:fe56fcdfea856aeab51c177738df48f93322d70cB.Preston<div class="wikitext"><p>When you talk of "janky" behaviours, it occurs to me they provide unintended hints about how these systems operate. In my youth, I wondered why a window would flicker when I dragged something over it. What was going when I launched an MS Windows game and the colours got all weird on the non-game parts of the screen? How did the later versions of Norton Utilities for DOS get an arrow-shaped mouse pointer moving pixel-by-pixel in text mode (and what was that hump between character cells about, and why wasn't it a blank column)? Many of my earliest programming projects were attempts to figure out such behaviour, and take advantage of it or find better ways to do it.</p>
<p>I guess this is related to what's sometimes called a "revealing error". I'm happy that computers are working better, but I never really noticed till now that we've kind of lost something. (There's an interesting blog called "revealing errors"—still online but not updated—with some examples of what we can infer from misbehaved technology.)</p>
<p>I wasn't using X till the late 1990s, when people were already starting to expect backing stores, so wasn't aware that it was a separate feature from "save-under" or that it was specified as a "best-effort" cache to help the server deal with memory pressure. Thanks for giving a bit more history. When I worked on memory-limited embedded systems, nobody questioned the need for backing stores, but sometimes we'd wonder whether our window could use one instead of two. Amusingly, by <a href="https://blurbusters.com/blur-busters-lagless-raster-follower-algorithm-for-emulator-developers/">bringing the "beam-racing" technology of the 1970s into the modern day</a>, it can give even better results than a naive doubled-buffered program, though the comparatively boring 1980s-style "draw during vblank" was good enough for our purposes.</p>
</div>2024-03-05T07:37:58ZBy B.Preston on /blog/unix/XRenderingVsWaylandRenderingtag:CSpace:blog/unix/XRenderingVsWaylandRendering:b9eb9386676e597a7788b08e55335c3ad76a3bbaB.Preston<div class="wikitext"><blockquote><p>In the days of 10 Mbit/second local area networks and also slow inter-process communication on your local Unix machine, [server-based rendering] was a relatively important part of both X's network transparency story and X's performance in general.</p>
</blockquote>
<p>An aspect you're overlooking is memory usage. X dates back to 1984, when the idea of spending something like 100K for each window was untenable. Thus, till around the late 1990s, graphics systems based around drawing were common—including MS Windows, QNX Photon, and many others. Programs would be told when it was time for them to draw (because previously-invisible portions of their windows became visible), and they'd be given direct access or protocol-mediated access to "the framebuffer". Delays in drawing would often be user-visible. (Colour palettes tended to result in visible artifacts too; younger readers may find the concept of web-safe colours interesting, though Wikipedia lacks any example of how bad things could look when "unsafe" colours were used.)</p>
<p>Now, when we have gigabytes of memory, it's no big deal to have two or three 8-megabyte buffers for each window. Even with double-/triple-buffering, "true colour", and higher resolutions, the space needed for window bitmaps increased much more slowly than system memory. So, "modern" graphics systems are based around buffers instead of drawing. Programs are expected to keep their buffers up to date, and often have no knowledge of whether (or to what extent) they're visible; the graphics server can grab data from those buffers as required, eliminating the flickering.</p>
<p>With 100-megabit networks, and especially gigabit, client-side rendering with X is often faster across these networks than server-side, because most X clients don't handle latency well—waiting for acknowledgement of each drawing command instead of doing it in batches. On a campus LAN the difference is hardly noticeable, but when latency gets above a few milliseconds, one's usually better off having X clients connect to a server on the same machine and using something like VNC to transport the bitmap data.</p>
</div>2024-03-04T18:24:41ZBy mappu on /blog/unix/XRenderingVsWaylandRenderingtag:CSpace:blog/unix/XRenderingVsWaylandRendering:4b4f4587287dcb06feb09840065c484102f4b520mappu<div class="wikitext"><blockquote><p>Substituting in a stand alone client side drawing library is probably not a small job, since I don't think any of them so far are built to be API compatible with the relevant X APIs.</p>
</blockquote>
<p>One example of such a library is `xlibe` (<a href="https://github.com/waddlesplash/xlibe">https://github.com/waddlesplash/xlibe</a>). This implements the Xlib API entirely in-process, there's no actual X11 server. Effectively, it's a drawing library.</p>
</div>2024-03-04T04:57:20ZBy Twirrim on /blog/tech/ServerCPUDensityAndRAMLatencytag:CSpace:blog/tech/ServerCPUDensityAndRAMLatency:bd825bf2b887e25544bd603771b2bd5a14612a82Twirrim<div class="wikitext"><p>NUMA is your biggest concern when it comes to RAM latency, and with increasing core counts, it's only going to get worse. NUMA has a <em>lot</em> of quirks to it that can dramatically influence performance.</p>
<p>Without going too deep in to the subject, but the cores in your system are grouped together into NUMA nodes, each node is directly attached to a particular subset of memory, and indirectly attached to the rest via the other nodes, paying the penalty of that extra hop between it and the memory. That adds noticeable latency to every request.</p>
<p>It can have some really significant impact. For example, Oracle has been exploring having ktext replicated in to each NUMA domain on arm64 (which in server class chips tends to be even more "NUMA"ish), <a href="https://lwn.net/Articles/956900/">https://lwn.net/Articles/956900/</a>, "[the patches] show a gain of between 6% and 17% for database-centric like workloads. When combined with userspace awareness of NUMA, this can result in a gain of over 50%."
Having to reach across to the other NUMA node to get to the executable code in the kernel turns out to be an expensive and common operation.</p>
<p>It only gets worse from there, for example the linux page cache isn't fully NUMA aware. I know of someone who tripped up on this benchmarking NUMA nodes. They thought they had two very different performance NUMA nodes in the system they were benchmarking. In reality, it turned out the mysql client library got cached in one NUMA node's memory, during the previous benchmark run, and so the calls for the functions exposed by the client library were having to go cross-NUMA!</p>
<p>CXL etc. that are in the pipeline will also make these kinds of concerns increasingly important, as they talk about CXL in terms of adding a NUMA node hop or two cost.</p>
</div>2024-03-04T02:26:14ZBy Fazal Majid on /blog/tech/ServerCPUDensityAndRAMLatencytag:CSpace:blog/tech/ServerCPUDensityAndRAMLatency:1ab99c59c7dcf2a246a477f7eb6d7d0c7b8c19d9Fazal Majidhttps://majid.info/<div class="wikitext"><p>NUMA has more of an impact. In my experience PostgreSQL performance is most correlated with the STREAMS benchmark and Amazon AWS’s biggest instances underperform some cheaper ones due to that NUMA penalty.</p>
</div>2024-03-03T12:37:19ZBy Fazal Majid on /blog/tech/ServersSpeedOfChangeDowntag:CSpace:blog/tech/ServersSpeedOfChangeDown:d31f5b4b475147f1057c72bcd7048d40c1c086a9Fazal Majidhttps://majid.info/<div class="wikitext"><p>AWS is raising efficiency with its Graviton series arm64 CPUs, and it’s easier to move functions as a service than legacy platform as a service workload. Still, when my former employer rebuilt its dockers on ARM, we got 30% cost savings.</p>
<p>Newer CPUs are also adding features like encrypted memory that protect VMs’ memory from snooping by the hypervisor, and thus allow sensitive industries like health care to move to the cloud.</p>
</div>2024-03-02T15:21:55ZBy Lars Windolf on /blog/sysadmin/PrometheusAbsentMetricsAndLabelstag:CSpace:blog/sysadmin/PrometheusAbsentMetricsAndLabels:6206cf28b650aed33d345e4025dde9b0afe1a586Lars Windolf<div class="wikitext"><p>This really helped me this week! Was looking for a solution of exactly this problem. Thanks!!!</p>
</div>2024-03-02T11:33:54ZBy sapphirepaw on /blog/tech/ServersSpeedOfChangeDowntag:CSpace:blog/tech/ServersSpeedOfChangeDown:e29df789abd441c23d9bdc7fc92e3c1d9ceb455esapphirepawhttps://www.sapphirepaw.org/<div class="wikitext"><p>AWS has reached a point where they are always touting "price efficiency" of new hardware generations. "Best price/performance ever!" when you boost prices 8% and performance 10%. It looks from the outside like they can't get higher performance without raising their own costs.</p>
<p>The floor of the "smallest configurable instance" never goes down, either. They filled in underneath <code>m*.large</code> with the <code>t*</code> families, which are CPU-throttled to emulate smaller slices of the underlying hardware.</p>
</div>2024-03-01T15:56:08ZBy edgewood on /blog/sysadmin/RsyncRecentDirectoryContentstag:CSpace:blog/sysadmin/RsyncRecentDirectoryContents:1357252b8b3214fa1ee924038a8a39806283f6a9edgewood<div class="wikitext"><p>Late comment, but I almost always use the rsync options <code>--itemize-changes</code> and <code>--dry-run</code> to ensure that the rest of the options do what I'm expecting, then drop <code>--dry-run</code> when I'm satisfied.</p>
</div>2024-03-01T00:57:27ZBy Anonymous on /blog/sysadmin/WhyNoMachineInventorytag:CSpace:blog/sysadmin/WhyNoMachineInventory:62ff7c7aef09245b04902cc9b50174e126d490f3Anonymous<div class="wikitext"><p>Just two words: Configuration Management.</p>
</div>2024-02-28T21:43:25ZBy Ivan on /blog/tech/OpenSourceCultureAndPublicWorktag:CSpace:blog/tech/OpenSourceCultureAndPublicWork:b82c48ca6bfba6129db8ca6168a6846f87ed8cafIvan<div class="wikitext"><p>Related might be <a href="https://rachelbythebay.com/w/2018/10/09/moat/">the choice to stay out of the community altogether</a>, although for not exactly these reasons.</p>
<p>There are forces in the community which try to address this problem, e.g. by <a href="https://allcontributors.org/docs/en/emoji-key">recognising the many ways to contribute which aren't limited to code</a> (although these technical choices are not for everybody). On the other end of the spectrum is the rare project like SQLite where <a href="https://sqlite.org/copyright.html">code is not that welcome and patches, if they are submitted, are most likely to be rewritten</a> for provenance and accountability reasons.</p>
</div>2024-02-26T07:01:35ZBy Mike Tancsa on /blog/sysadmin/SSHBruteForceAttacksAbruptlyDowntag:CSpace:blog/sysadmin/SSHBruteForceAttacksAbruptlyDown:179a541f0728148712dca9abedb6225559867f9fMike Tancsa<div class="wikitext"><p>My network (couple of /18s worth of ip space) gets hit in fits and starts as well. I also wonder if companies like Shodan have created a secondary market for players to constantly scan the internet and sell their results.
Anyways,
<a href="https://www.dshield.org/data/port/22">https://www.dshield.org/data/port/22</a>
doesnt show any overall trend changes as of late.</p>
</div>2024-02-24T23:16:41ZBy Anonymous on /blog/sysadmin/SSHBruteForceAttacksAbruptlyDowntag:CSpace:blog/sysadmin/SSHBruteForceAttacksAbruptlyDown:f1a51b55597b80079c1c0f8f2bf1c12d8cf0754fAnonymous<div class="wikitext"><p>Perhaps someone - yet unannounced - took a (mayor) botnet offline ?</p>
</div>2024-02-23T21:56:59ZBy Phong on /blog/tech/DesktopECCOptions2024tag:CSpace:blog/tech/DesktopECCOptions2024:122b816c64f9bdbf53d1b539996ea1b1d1b46b5ePhong<div class="wikitext"><p>I have the Asus W680 motherboard mentioned on the Fediverse with an i7-13700K and it does appear to properly support ECC (at least as Windows 11 can tell). I have not had any problems so far running it as my main desktop.</p>
</div>2024-02-23T20:54:03ZBy jlin on /blog/sysadmin/SSHBruteForceAttacksAbruptlyDowntag:CSpace:blog/sysadmin/SSHBruteForceAttacksAbruptlyDown:f17e211db730158ed8f4e1ddd1dc8afb96825bfdjlin<div class="wikitext"><p>Could it be something like packet filtering at the upstream?</p>
</div>2024-02-23T09:36:48ZBy kim s on /blog/linux/NetworkManagerDoesNotSharetag:CSpace:blog/linux/NetworkManagerDoesNotShare:2d41000efe9bf90477e3b8079bbc26a2148221ddkim s<div class="wikitext"><p>I've found Network Manager to be annoyingly inflexible. For example, my wired interface was configured as a bridge (for virtual machines), and it refused to touch that. And it didn't work well for DHCP interfaces with some static settings, and always wanted to rewrite resolv.conf. Also, corporate policy prohibited network passwords being saved on disk, but I don't recall being able to get it to prompt for but not permanently store the password.</p>
<p>For wi-fi, it turns out that it's not difficult at all to script wpa_cli. A reasonably simple shell script can send a GET_PASSPHRASE request to gpg-connect-agent and feed the result to wpa_cli (but then getting it to run dhclient or whatever takes a bit of extra configuration as root). The authentication-free wired network was just done using Debian's "if-up" stuff, which is... kind of okay, if one remembers to put " || true" after the pre-up/post-up commands. I'd prefer to have something like Network Manager with good support for hooks, scripts, and similar customization, so I could fully configure a network without being root. I haven't found anything yet.</p>
</div>2024-02-21T18:14:23ZBy Andrew on /blog/linux/NetworkManagerDoesNotSharetag:CSpace:blog/linux/NetworkManagerDoesNotShare:670615adc7b66794e522df3772f883e770239914Andrew<div class="wikitext"><blockquote><p>I suspect that NetworkManager applies this restriction to all VPNs and other layered forms of networking, such that you can only run a NetworkManager managed VPN over a network interface that NetworkManager is controlling.</p>
</blockquote>
<p>I don't think that's true. I don't do things this way anymore, but I used to have eth0 managed by ifupdown (and marked "unmanaged" in NM), while using NM-managed VPNs (because they were only transiently connected and that was the easiest way to do it). It never complained.</p>
<p>PPPoE may be different for silly reasons, I have no experience with it.</p>
</div>2024-02-21T07:17:43ZBy Emmy on /blog/web/TLSCertsSomeStillManualtag:CSpace:blog/web/TLSCertsSomeStillManual:45121fdf1edf52a2d2923beee4e0a70a2651d5feEmmy<div class="wikitext"><p>Browsers seem to have decided on 398 days for the maximum duration for TLS certificates. But I couldn't find an official explanation on why that number exactly. One Stack Overflow answer says it's 366 + 31 + 1 (leap year + 1 month + extra 24 hours for time zone differences).</p>
<p><a href="https://blog.mozilla.org/security/2020/07/09/reducing-tls-certificate-lifespans-to-398-days/">https://blog.mozilla.org/security/2020/07/09/reducing-tls-certificate-lifespans-to-398-days/</a></p>
<p><a href="https://chromium.googlesource.com/chromium/src/+/refs/heads/main/net/docs/certificate_lifetimes.md">https://chromium.googlesource.com/chromium/src/+/refs/heads/main/net/docs/certificate_lifetimes.md</a></p>
<p><a href="https://support.apple.com/en-us/102028">https://support.apple.com/en-us/102028</a></p>
</div>2024-02-19T04:38:58ZBy Cristina on /blog/tech/DesktopECCOptions2024tag:CSpace:blog/tech/DesktopECCOptions2024:eff7ee86602c94bb3ad37c6948f9b20caba36d6cCristina<div class="wikitext"><p>A little over a decade ago, going with a Xeon and ECC instead of a desktop Intel chip added maybe 2 or 3 hundred dollars to the system cost, and I built such a system for under $1500. Now, it seems Xeon has nothing much cheaper than Threadripper. I see some Sapphire Rapids Xeon parts (one generation behind) in the $600-$1000 range.</p>
<p>For a while, Intel also supported ECC on some desktop i3 chips, such as i3-8100. The rumour was that they did this to target embedded markets, unlike the otherwise-higher-end i5 through i9 which lacked ECC. I don't know whether such support still exists.</p>
<p>The "workstation" motherboards do, in general, seem to be closer to what Chris is looking for. For example, I see several with three to seven M.2 slots, dual ethernet ports (or triple, but it looks like the third is usually reserved for BMC), 15+ USB ports, 6 or 8 SATA, even up to seven full-speed full-width PCIe slots. The Asus Pro WS Sage SE boards—W790E or WRX80E—are examples for those who don't mind spending $1400. Or the Gigabyte TRX50 Aero D is $400 cheaper but with only 3 PCIe slots.</p>
<p>The aforementioned 3 boards all seem to need RDIMMs. Personally, I'd probably go for unofficial DDR5 ECC UDIMM support according to web forum posts, maybe with Ryzen 8000G Pro APUs when available. Or catch the trailing edge of W680 boards and CPUs as they drop in price, and add an ethernet card if more than one port is needed. But I'd love to see that hypothetical "W780". As Linus Torvalds has ranted, ECC shouldn't be a "premium" feature. Researchers have found via experiments like "bit-squatting" that memory errors do occur in the wild, often enough to be of concern.</p>
<p>Anyway, thanks for this post, and please keep us updated if you manage to build a low-cost system with working ECC.</p>
</div>2024-02-17T18:58:53ZBy Cristina on /blog/tech/DesktopECCOptions2024tag:CSpace:blog/tech/DesktopECCOptions2024:309d137e492a4712b93ba5f3495f9fd964b4fc3aCristina<div class="wikitext"><p>Jonathan, the Threadripper (TR) is considered a workstation board rather than a desktop board per se. As Chris wrote, "The traditional option to getting ECC RAM support (along with a bunch of other things) was to buy a 'workstation' motherboard…"; that sentence was about Xeon but is just as true for TR.</p>
<p>Threadripper chips and boards support ECC officially, and TR might be the reason for desktop Ryzens supporting ECC at all: at least in the original version, Ryzen and TR used the same dies (TR having been a "spare time" project). But it looks hard to get a new current-generation model for less than 2 or 3 thousand dollars (I do see a previous-generation 5955WX—literally just one, in Ottawa—for $1000 at Memory Express). The motherboards are not cheap either. While a nice option for those who can get an employer to pay, this decision will probably double or triple the cost of a system.</p>
</div>2024-02-17T16:59:00ZBy Chris Siebenmann on /blog/tech/DesktopECCOptions2024tag:CSpace:blog/tech/DesktopECCOptions2024:3b27af5ffc920933f2697d3e5e1b4beae39496d8Chris Siebenmann<div class="wikitext"><p>Oops yes, Threadripper class AMDs do support ECC and you can get
motherboards for them and build your own desktop that way. For some
reason I always push this out of my mind as a crazy option, but it's
probably not more so than a Xeon-based build.</p>
</div>2024-02-17T16:54:58ZBy Thomas on /blog/tech/DesktopECCOptions2024tag:CSpace:blog/tech/DesktopECCOptions2024:ad8c58678265a9a5702199bb54c1f1f5690c3f6eThomas<div class="wikitext"><p>I’ve got an ASRock Rack AM5 1U system that I’m quite happy with - a ryzen 9 7900 + 128G of RAM makes a really nice small rack server, and uses <100W nearly all the time.</p>
</div>2024-02-17T12:54:04ZBy Jonathan on /blog/tech/DesktopECCOptions2024tag:CSpace:blog/tech/DesktopECCOptions2024:d374176f476976abad9a0096e6951a2692e83eb2Jonathanhttps://jmtd.net<div class="wikitext"><p>Pretty sure the Threadripper class AMDs support ECC. I have a Lenovo P620 workstation with TR and ECC.</p>
</div>2024-02-17T11:42:00ZBy Verisimilitude on /blog/spam/AntiSpamIsAllHeuristicsNowtag:CSpace:blog/spam/AntiSpamIsAllHeuristicsNow:c128902d6b1b3065d55fa5cbe224a59561e8f8a6Verisimilitudehttp://verisimilitudes.net<div class="wikitext"><p>I disagree, for more than two reasons. As e-mail isn't time-sensitive, it would be feasible to have proof-of-work requirements that last into the tens of minutes or hours. This basically kills the drive-by spam. In a system without an authority, the idea of legitimacy loses most of its meaning. Regardless, Hashcash could be used as a way to get a sender address into a whitelist, and no address in a whitelist would need Hashcash to be accepted. Hashcash could be used entirely as a signal for automatic categorization like this. Free Software mailing lists, and similar such lists, could be collected into some whitelist for interested parties.</p>
<blockquote><p>Hashcash is a non-starter in anything like even a traditional, pre-spam Internet email environment, much less today's non-spam email environment.</p>
</blockquote>
<p>Explain why. I'm not seeing it. The main opposition to Hashcash are the aforementioned companies which want to send their spam to everyone.</p>
</div>2024-02-16T22:47:15ZBy Chris Siebenmann on /blog/spam/AntiSpamIsAllHeuristicsNowtag:CSpace:blog/spam/AntiSpamIsAllHeuristicsNow:a5b043c73a2dd0358b3d2d9a17c75b1ee8a47d5fChris Siebenmann<div class="wikitext"><p>I disagree, for two reasons. First, <a href="https://utcc.utoronto.ca/~cks/space/blog/spam/EmailChargingEffects">charging for email in general is
not going to stop spam, although it will change what sort of spam you
get</a>. This includes Hashcash, especially now that
you can rent compute capacity as you need it (so people who want to send
out a marketing email campaign can literally pay for the Hashcash costs,
were they exist). Second, <a href="https://utcc.utoronto.ca/~cks/space/blog/spam/EmailChargingEffects">Hashcash harshly penalizes legitimate senders
of significant amounts of email</a>, including mailing
lists, who see their compute needs and thus costs go up drastically.
Hashcash is a non-starter in anything like even a traditional, pre-spam
Internet email environment, much less today's non-spam email environment.</p>
<p>(Plus, active criminal spammers have plenty of compute capacity they can
rent for cheap, cf botnets for hire.)</p>
</div>2024-02-16T20:31:29ZBy Verisimilitude on /blog/programming/LanguageNichesIItag:CSpace:blog/programming/LanguageNichesII:f1869b905e5c0d17d7360c64e970c4cc1ba57e2aVerisimilitudehttp://verisimilitudes.net<div class="wikitext"><p>This categorization seems incomplete to me. I'd believe Common Lisp to be in the first niche, and see no place for APL in these categories. This new niche is silly, to put it nicely.</p>
<blockquote><p>In the existing niches, Rust has made major inroads in the 'low level with strong control' niche.</p>
</blockquote>
<p>Every time I see someone mention Rust, it's as if in total ignorance of Ada, which has been doing all of this and more for decades. Of course, Ada isn't so popular in comparison, but it's used for real work, where people die, or the wrong people die, if something goes wrong.</p>
<p>Popularity is irrelevant.</p>
</div>2024-02-16T18:32:15ZBy Verisimilitude on /blog/spam/RemoveUserAgentHeadertag:CSpace:blog/spam/RemoveUserAgentHeader:52b01ec4511b4bcaf03ba4eabdb9460926647e1dVerisimilitudehttp://verisimilitudes.net<div class="wikitext"><p>I used to use a fake value for this field for one address, the name of the WWW client even though I used a real client, but agree, and removed it long ago. It made no observable difference.</p>
</div>2024-02-16T18:13:57ZBy Verisimilitude on /blog/sysadmin/CustomizationSensibleLimitstag:CSpace:blog/sysadmin/CustomizationSensibleLimits:1ab7d49e69b96a25eed5277ea3c7f5de9413b8b1Verisimilitudehttp://verisimilitudes.net<div class="wikitext"><p>Ideally, programs should be small enough they can be understood and directly customized without the need for a special configuration format or language. While not what I mean, I've taken an Emacs mode and directly modified it, to load my modified version instead, because I couldn't figure out how to make the particular change otherwise. An ultimate form of this is designing and creating one's own tools, and slowly replacing Emacs piecewise like this; it's still my main writing tool, however.</p>
</div>2024-02-16T18:00:46ZBy Thomas Adam on /blog/unix/XOffscreenWindowsUsetag:CSpace:blog/unix/XOffscreenWindowsUse:c45b9146e6647ea6ee4938ab7718b1dba91b00ceThomas Adam<div class="wikitext"><p>Uh huh.</p>
<p>It's important to draw the distinction between pages and desks as fvwm sees them.</p>
<p>With pages (and hence the current desk -- there's always at least one desk defined), all windows on that desk are mapped.</p>
<p>With multiple desks, windows on other desks other than the current desk being viewed are unmapped. In this way, it means the effect of viewing all windows across pages for that desk makes sense, without other windows from other desks "interfering". This is how fvwm2 works, and also fvwm3 with "DesktopConfiguration global".</p>
<p>In fvwm3 however, if one uses "DesktopConfiguration per-monitor", this rule breaks that model -- if were to use the "Scroll" command to move around pages incrementally, you'll find those window locations overlap windows on other desks.</p>
</div>2024-02-16T17:59:40ZBy Chris Siebenmann on /blog/sysadmin/GrafanaLokiStartupWALReplayIssuetag:CSpace:blog/sysadmin/GrafanaLokiStartupWALReplayIssue:7d9560b74b04416c1979372116a27feb222e696dChris Siebenmann<div class="wikitext"><p>There are a number of issues with Victorialogs today, starting with how
they explicitly say "it isn’t recommended to migrate from existing
logging solutions to VictoriaLogs Preview in general cases yet". Beyond
that it still relies on Loki's Promtail for shipping logs from systemd
and syslog, so we would only be half moving away from Loki, and it also
doesn't appear to have any integration with Grafana. Their current
documentation also says it's missing features from LogQL, some of which
we make significant use of in current queries.</p>
<p>Victoriallogs may someday be a complete, self-contained full replacement
for Loki, but it's not currently one.</p>
</div>2024-02-16T17:18:51ZBy Chris Siebenmann on /blog/linux/ZFSPoolTXGsInformationtag:CSpace:blog/linux/ZFSPoolTXGsInformation:45ee0d0fac49b34cff4bb148640d25293fa2c0b5Chris Siebenmann<div class="wikitext"><p>I've updated the entry with a note about this. Writing what I did is
especially foolish considering that on the machine I took the txgs
example from, I had earlier seen commits in state S in the process of
syncing out.</p>
<p>(I was watching the txgs file because this particular machine was having
some write load problems, where it turns out txgs were sometimes taking
more than 30 seconds to commit. That sort of thing would definitely give
you at least one txg in a state between open and committed.)</p>
</div>2024-02-16T15:45:49ZBy Verisimilitude on /blog/unix/UnixAPIAndCRuntimetag:CSpace:blog/unix/UnixAPIAndCRuntime:4f2930e45c41f0a15b6175096b892f16d18fe115Verisimilitudehttp://verisimilitudes.net<div class="wikitext"><p>Unfortunately and despite contrary claims, it is effectively so. Avoiding it is necessary, difficult, and poorly-documented. Pull in one speck, and the entire pig wants in the house.</p>
<p>However, attempting to force the C language on us is their undoing. Those text-replacement macros and other nonsense are too hard to change and, because the only value in UNIX is what already runs on it, they never will be changed. This makes the initial effort to circumvent the garbage worthwhile, because they dare not make large changes, even though they believe them to be possible.</p>
</div>2024-02-16T08:21:10ZBy Verisimilitude on /blog/programming/UnmaintainedCodeHugeValuetag:CSpace:blog/programming/UnmaintainedCodeHugeValue:0e5a78e9c0b28fa55036deb5977363d48e81186bVerisimilitudehttp://verisimilitudes.net<div class="wikitext"><p>Finished software has everlasting value. Most unmaintained software isn't finished. Regardless, it's perfectly reasonable to state that code only works with some particular standard of a language, and for future standards of that same language to be incompatible with it. This is how Ada works, and she works more than fine that way.</p>
</div>2024-02-16T05:27:11ZBy valyala on /blog/sysadmin/GrafanaLokiLogcliNotestag:CSpace:blog/sysadmin/GrafanaLokiLogcliNotes:0c6f98bde5b0c64b66e262664138e9a7aae0a619valyala<div class="wikitext"><p>You definitely need to read <a href="https://docs.victoriametrics.com/victorialogs/querying/#command-line">https://docs.victoriametrics.com/victorialogs/querying/#command-line</a> and stop hurting yourself with logcli :)</p>
</div>2024-02-16T00:21:07ZBy Verisimilitude on /blog/tech/IPv6IsTheFuturetag:CSpace:blog/tech/IPv6IsTheFuture:ac6691bdb9fe31f8efd0b7b600f06dda8fd12f9cVerisimilitudehttp://verisimilitudes.net<div class="wikitext"><p>Nearly eight years later, the situation for IPv6 still looks bad. The only reason it's made the progress it has is due to so-called <em>smart</em>phones, which is another way of saying large corporations manage things, and the people have little to no say.</p>
<p>It's far more likely IPv6 is replaced in favour of something else. IPv4 will undoubtedly outlive IPv6, and I'd wager on that. Everything important is still available over IPv4, whereas nothing important is only available over IPv6. I'll continue to design interfaces that only allow IPv4.</p>
</div>2024-02-16T00:15:01ZBy valyala on /blog/sysadmin/GrafanaLokiStartupWALReplayIssuetag:CSpace:blog/sysadmin/GrafanaLokiStartupWALReplayIssue:0bcd58b73329aeb53c067f2b0aa64793b7c1cbc5valyala<div class="wikitext"><p>Why you still use Loki and don't switch to something more reliable and resource efficient? For instance, victorialogs . It is open source, it is easy to setup and operate - just a single small statically linked binary without external dependencies. It is much faster at querying than Loki. And it integrates well with traditional Unix commands - see <a href="https://docs.victoriametrics.com/victorialogs/querying/#command-line">https://docs.victoriametrics.com/victorialogs/querying/#command-line</a></p>
</div>2024-02-16T00:10:59ZBy Verisimilitude on /blog/unix/BindingOutgoingSocketstag:CSpace:blog/unix/BindingOutgoingSockets:0b03016cf6bc10a8a5732932a101918915a87522Verisimilitudehttp://verisimilitudes.net<div class="wikitext"><p>I learned this when writing a <a href="http://verisimilitudes.net/2022-08-22">UDP interface in Ada</a>, which unfortunately required POSIX Sockets, but I sequestered away that in <a href="http://verisimilitudes.net/2022-04-05">a separate package</a> which the proper interface used.</p>
<blockquote><p>I'm sure there's some advantage to this 'bind before connect' approach, but I'm honestly hard pressed to think of any.</p>
</blockquote>
<p>As am I. It could be useful to ensure that system resources are allocated ahead of time, but for TCP in particular it's best to only allow full connections, with both ends already known, over half connections with only one end known. I'm in the midst of designing a TCP interface in Ada, and it should be perfectly feasible to make <code>connect</code> implicit, which is almost a shame, since I had a great name for it, <code>meet</code>.</p>
</div>2024-02-16T00:00:11ZBy Verisimilitude on /blog/spam/AntiSpamIsAllHeuristicsNowtag:CSpace:blog/spam/AntiSpamIsAllHeuristicsNow:2ffaf38e43c189d7797138cf22a5c67a8206456aVerisimilitudehttp://verisimilitudes.net<div class="wikitext"><p>The only real solutions I see to this are whitelists, blacklists, and Hashcash. The last has been ruthlessly stopped by large corporations in favour of <em>reputation</em>, because these large corporations all want to send spam. Personally, I believe accidentally missing an e-mail to be unacceptable. These same large corporations like to have their servers lie about accepting e-mails, because they want to destroy what they can't control.</p>
<blockquote><p>It's because anti-spam isn't working against a blind natural phenomenon; instead, anti-spam is engaged in an iterated game against human driven spam.</p>
</blockquote>
<p>Let's not forget that proof-of-work is <em>bad for the environment</em>, that message brought to us by the same corporations that callously waste resources and produce mountains of plastic.</p>
</div>2024-02-15T23:01:21ZBy Verisimilitude on /blog/unix/WhyNoStandardUnixGUIstag:CSpace:blog/unix/WhyNoStandardUnixGUIs:6819034a0747c246eb3b01ac12c357eaae83983fVerisimilitudehttp://verisimilitudes.net<div class="wikitext"><blockquote><p>Because I am a jaundiced and cynical person, I think that the real reason GUIs were not standardized is that they could not be, because they were being viciously fought over by the vendors all through the period when Unix standards were being created.</p>
</blockquote>
<p>Was this meant to read <em>jaded</em>? Anyway, GUIs certainly could have been standardized, but UNIX isn't a graphical operating system, and the lack of standardization makes this very clear.</p>
<blockquote><p>GUIs are big and complex, which means that they are expensive to develop, which means that everyone who had one wanted money to let other people use it (well, their code for it).</p>
</blockquote>
<p>I can agree they're big, complex, and expensive, under UNIX. I've not read any such thing about Xerox PARC's GUI, but hey, they only invented the modern GUI while Ritchie and Thompson were still playing with teletypewriters, and it's hardly the fault of UNIX that it still can't do something people did half a century ago; that's just how UNIX is.</p>
<blockquote><p>There was also a technical perspective, which I would characterize as saying that GUI standardization was vastly premature at the time because we didn't know enough to make a good, enduring GUI standard, in either appearance or API.</p>
</blockquote>
<p>It's fair to say that a low-level GUI API is an oxymoron. It's easy when one lifts himself out of the tarpit of managing individual bits, and unreasonably difficult otherwise.</p>
<blockquote><p>It may even be premature now, given that all of the de facto standard GUIs today are still evolving and changing.</p>
</blockquote>
<p>Well, nearly fourteen years later, it's safe to say the situation is even worse. Now people seriously use possibly the worst possible system for designing a GUI: HTML, CSS, and JavaScript.</p>
</div>2024-02-15T22:24:17ZBy Verisimilitude on /blog/sysadmin/ReportConfigFileLocationstag:CSpace:blog/sysadmin/ReportConfigFileLocations:7ead682dd24ca48bfa0b1ed1398d9349ea02a886Verisimilitudehttp://verisimilitudes.net<div class="wikitext"><blockquote><p>if a program uses a configuration file (or several), it should have an obvious command line way to find out where it expects to find that configuration file.</p>
</blockquote>
<p>This sounds an awful lot like a standard mechanism for configuration, which UNIX lacks.</p>
<blockquote><p>Ideally I'd like to avoid scanning all the way through the manual page or other documentation for the program to find out, because that's slow and annoying.</p>
</blockquote>
<p>I don't think MicroSoft Windows has this problem, but I regularly read people bitching about the central configuration mechanism it has, whatever its name is. Anyway, this is the fun of using <em>convention</em> for everything, nothing actually works.</p>
<p>Personally, I've grown to believe configuration files are stupid, and everything I design lacks them. Ideally, every program is so small that it can be modified directly for such configurations.</p>
</div>2024-02-15T21:37:11ZBy Teresa Jones on /blog/programming/GoKeepsConstantVariablestag:CSpace:blog/programming/GoKeepsConstantVariables:6f68422bca44b6db2c2baea961d0dc73f9a1f2c9Teresa Jones<div class="wikitext"><p>"var doThing bool" is a global var not a const !!!!</p>
<p>The clue is in the name.</p>
<p>"const doThing bool" would be a const.</p>
</div>2024-02-15T18:34:35ZBy Miksa on /blog/linux/MyMachineDesires2024tag:CSpace:blog/linux/MyMachineDesires2024:32b8816a341fd9841c9c08e638aea81af04b4d93Miksa<div class="wikitext"><p>@gmit, I suspect that is practically a non-issue. NVMe has so much bandwidth and there is only so much that humans or CPUs can process. What work could CPU do for extended time at 2+ gigabytes per second?</p>
<p>If you have a M.2 and 100Gb network card sharing PCIe bandwidth and you are transferring your full drive they can probably slow each other down, but not enough to be a problem.</p>
<p>PCIe 4.0 is becoming common and that provides M.2 slot with 8.0GB/s of bandwidth and I think fastest M.2 drives still top out around 5GB/s. And you can always upgrade to PCIe 5.0 if that is a problem.</p>
</div>2024-02-14T17:01:51ZBy Arnaud Gomes on /blog/linux/ZFSPoolTXGsInformationtag:CSpace:blog/linux/ZFSPoolTXGsInformation:0c95492ec281c21606aa7dd084ed2eaee7f60c67Arnaud Gomes<div class="wikitext"><p>Just to contradict you, I have just seen a machine with three non-committed TXGs, in states S, W and O from the oldest to the newest. This is during a <code>zfs recv</code> under quite a heavy I/O load.</p>
<pre>
-- A
</pre>
</div>2024-02-14T14:36:20ZBy mappu on /blog/linux/HomeBackupPlans2024tag:CSpace:blog/linux/HomeBackupPlans2024:cd8dd3ebec6ad165e0e7cdc7a2562c5f63952c8dmappu<div class="wikitext"><p>I advise doing a little more than this. What if there's a physical disaster or a burglary at your home?</p>
<p>The 3:2:1 is you should have three copies - on two different mediums - and one of them should be offsite,</p>
<p>Check out kopia, borg, or restic for something that can do encrypted, deduplicated, incremental-forever backups to cheap cloud storage (Wasabi is 7USD/TB, Backblaze B2 is 6 USD/TB).</p>
</div>2024-02-13T23:33:59ZBy Chris Siebenmann on /blog/programming/Go122TypesAliasAndCompatibilitytag:CSpace:blog/programming/Go122TypesAliasAndCompatibility:a70e7ece884e3cf28b3d4e74156e9d616571096aChris Siebenmann<div class="wikitext"><p>Go doesn't have true enum / sum types as I understand them, and normal
interfaces are explicitly non-exhaustive (in that anyone can make a type
that implements an interface). However, I think even an exhaustive enum
type and support forcing exhaustiveness on things dealing with it wouldn't
truly help here; it would just change a runtime error to a compile time
error, not eliminate it.</p>
</div>2024-02-13T14:20:22ZBy Arash S.A.H. on /blog/linux/HomeBackupPlans2024tag:CSpace:blog/linux/HomeBackupPlans2024:e5ce714f4011772d22150416387ab589be4b8e0bArash S.A.H.<div class="wikitext"><p>I use an external HDD with a systemd-timer which runs a systemd-service which contains a short hand-made “rsync” script. Timer runs once a day... So any time that I plug it in, it starts syncing in background of my system. Using cgroups options of systemd-service I can be sure that it doesn't consume any CPU and I/O more than What I desire. So at the end of the story, the only thing that I need is to remember plugging my HDD once in a while.</p>
</div>2024-02-13T12:51:28ZBy arash.sah@tuta.io on /blog/unix/EdVersionsDifferencestag:CSpace:blog/unix/EdVersionsDifferences:9fca340ac5485d51a8aa47d90fcb839c541a7ab0arash.sah@tuta.io<div class="wikitext"><p>May you wonder, but believe it or not, I'm just a barista, working most of my time at my cafe! Not even a native English speaker! With all that being said, just due to my special interest to computer stuff and *x systems, started to use “ed” as my only text editor for two weaks! For editing everything. From small files to large ones! And it was a hard time, at first. But God... it actually did teach me all about extended regex. And then I have decided to give “ex” editor a try. I thought that they are similar... But no, there are many differences between them. In particular, they have different regex and line addressing. Started googling about this difference and I just came upon your post... Thanks for all the time you spent to write this. </p>
<p>Much Love and Regards</p>
</div>2024-02-13T09:22:01ZBy lilydjwg on /blog/programming/Go122TypesAliasAndCompatibilitytag:CSpace:blog/programming/Go122TypesAliasAndCompatibility:269786c78930bf0fd037fe146974c6593efa3717lilydjwg<div class="wikitext"><blockquote><p>Getting complete backward compatibility requires more or less everything about an exposed API to be frozen.</p>
</blockquote>
<p>Or it requires the language to express incompleteness, like marking an enum to be non-exhaustible---wait, does Go have enum / sum types?</p>
</div>2024-02-13T08:16:57ZBy Verisimilitude on /blog/programming/Go122TypesAliasAndCompatibilitytag:CSpace:blog/programming/Go122TypesAliasAndCompatibility:e82a1ebeb1549548b9a67c5d844cb8b3625abb3fVerisimilitudehttp://verisimilitudes.net<div class="wikitext"><p>I don't see what's so famous about it. The very first sentence is a contradiction to me, describing de jure but not de facto backwards compatibility. Common Lisp and Ada programs still work identically, decades later.</p>
<blockquote><p>When Go introduces a new concrete type that can be returned as a Type value, those previously exhaustive type switches stop being exhaustive; there's a new concrete type that they're not prepared to handle. This could cause various problems in actual code.</p>
</blockquote>
<p>That sounds like backwards incompatibility to me.</p>
<blockquote><p>And Go has no way of requiring type switches to be exhaustive, so such code would still build fine but malfunction at runtime.</p>
</blockquote>
<p>That's amusing to me. I've heard so much about how randomizing iteration order is so clever and important, but then they don't do something simple like this which Ada has been doing since the beginning.</p>
<blockquote><p>What this shows is that true long term backward compatibility is very hard, and it's especially hard in an area that is inherently evolving, like exposing information about an evolving language.</p>
</blockquote>
<p>It's not hard in the least. Ada does it just fine. Ada 1995 code will still work just fine, but a man can always choose Ada 2005 or 2012 for their later additions. Clearly, pretending to have just one version of the language is stupid.</p>
<blockquote><p>Getting complete backward compatibility requires more or less everything about an exposed API to be frozen, and that generally requires the area to be extremely well understood (and often pushes towards exposing very minimal APIs, which has its own problems).</p>
</blockquote>
<p>Alternatively, it requires careful thought.</p>
</div>2024-02-12T22:37:30ZBy Miksa on /blog/linux/HomeBackupPlans2024tag:CSpace:blog/linux/HomeBackupPlans2024:a44e297a2367ffb369a3fbf9afdcc05aa9ff22adMiksa<div class="wikitext"><p>My solution to this problem was to get a one large harddrive and install it to my work desktop at the university. Every day when I log in to work I run a rsnapshot job that connects to my home server. This might work for you too and it nicely solves the problem of having an offsite backup that avoids the hassle of carrying harddrives and is also online without needing to buy a cloud backup service.</p>
</div>2024-02-12T15:13:51ZBy Joseph on /blog/programming/Go122TypesAliasAndCompatibilitytag:CSpace:blog/programming/Go122TypesAliasAndCompatibility:9c0d77dd7a6362a24ee5787ce9b724eaa293a103Joseph<div class="wikitext"><p>Well, they really want type aliases for the iterator support being added in 1.23. They want to alias seq and seq2. So I expect that break in 1.23. Given how popular iterators will be, I expect people to be upgrade to 1.23 very quickly as well.</p>
</div>2024-02-12T04:02:20ZBy Blissex on /blog/linux/HomeBackupPlans2024tag:CSpace:blog/linux/HomeBackupPlans2024:02f3fd59884efadf3f77b22a0cd357683f10d4a1Blissex<div class="wikitext"><p>I think that a bit more effort is needed: I am currently "between desktops" but when I had a desktop I had 2TB drives duplicated nightly to 2 TB "alternate" drives (not mirrored, as mirrors can replicate problems too).
But I also had a rotation of 2x3 TB external using a SATA dock, done weekly.</p>
<p><a href="https://www.sabi.co.uk/blog/anno05-4th.html?051227b#051227b">https://www.sabi.co.uk/blog/anno05-4th.html?051227b#051227b</a>
<a href="https://www.sabi.co.uk/blog/0906Jun.html?090607#090607">https://www.sabi.co.uk/blog/0906Jun.html?090607#090607</a></p>
<p>Currently I have a cheap laptop upgraded with a single 4TB M.2 SSD, and I have for backup 2x external 4TB low-end SSD M.2 in a USB-C small case and 2x 4TB 2.5in USB3 HDDs, and I backup the internal 4TB to the USB-C SSDs twice (or more) a week, to one USB-3 disk every week, to the other disk every month.</p>
<p>The backup via RSYNC (I used to use 'dd' but with Btrfs duplicating volume ids is very risky) to the USB-C SSDs is very fast (huge IOPS on both sides) so I tend to do it fairly reliably.</p>
</div>2024-02-11T21:54:08ZBy David Magda on /blog/linux/HomeBackupPlans2024tag:CSpace:blog/linux/HomeBackupPlans2024:b3f353235a7a60caec1cb2825359f21b4a681e9aDavid Magdahttp://www.magda.ca/<div class="wikitext"><blockquote><p><em>ZFS's read only snapshots will insure that I can't accidentally damage the backup copies, and I can scrub the HDD's ZFS pool periodically as insurance against disk corruption.</em></p>
</blockquote>
<p>Starting with OpenZFS 2.2, any corruption found could also be fixed:</p>
<blockquote><p><em>Corrective "zfs receive" (#9372) - A new type of zfs receive which can be used to heal corrupted data in filesystems, snapshots, and clones when a replica of the data already exists in the form of a backup send stream.</em></p>
</blockquote>
<ul><li><a href="https://github.com/openzfs/zfs/releases/tag/zfs-2.2.0">https://github.com/openzfs/zfs/releases/tag/zfs-2.2.0</a><p>
</li>
<li><a href="https://github.com/openzfs/zfs/pull/9372">https://github.com/openzfs/zfs/pull/9372</a></li>
</ul>
<blockquote><p><em>In theory, what I should do to back up my home desktop is fairly straightforward. I should get one or two USB hard drives of sufficient size, then periodically connect one and do a backup to it (probably using tar</em> […]</p>
</blockquote>
<p>What I've found to be useful is to have a HDD dock (station) so I can just plug in any old drive and have it mount and then do a backup / clone of my data. I have a reminder in my personal calendar that pops up to remind me to regularly swap drives. It's a bit of a fixed cost, but over time as my backup needs have grown I don't to buy a 'special' USB drive, but any old hard drive with a SATA port (and docks that support M.2/U.2 are also available).</p>
</div>2024-02-11T12:33:12ZBy gmit on /blog/linux/MyMachineDesires2024tag:CSpace:blog/linux/MyMachineDesires2024:47f7b2a35f284c397746a3735e53aa50c4b0bf3bgmit<div class="wikitext"><p>You should avoid having SSDs on PCIe lanes going through the chipset as it's slower and could introduce various problems for other peripherals if you spend the whole bandwidth.</p>
<p>I've personally gone with Threadripper. Expensive, but everything works as it should.</p>
</div>2024-02-10T15:31:14ZBy Ian Z aka nobrowser on /blog/unix/XOffscreenIconMistaketag:CSpace:blog/unix/XOffscreenIconMistake:db628580565b9da4b42ed77ecf4f87d89ab9e4e5Ian Z aka nobrowser<div class="wikitext"><p>If you use the "server" feature of Emacs you can always reconnect by opening another client window, or even another session in a terminal.</p>
</div>2024-02-09T19:58:46ZBy Edward R on /blog/linux/MyMachineDesires2024tag:CSpace:blog/linux/MyMachineDesires2024:7a485d83b30db21c628a094f5c0da2a0a8622591Edward R<div class="wikitext"><p>In my view, some of those requirements are getting pretty esoteric. The combined number of PCIe and M.2 slots tends to total 7, at most. 6 SATA ports is doable, but may "steal" PCIe lanes from one of the slots unless it's a "workstation" board—ThreadRipper systems have a lot of lanes. If the third M.2 slot would only be needed rarely, for failure recovery or upgrades, I'd consider using an M.2-to-USB adapter and dealing with the less-than-optimal performance till the data's been cloned and it's convenient to power down for proper installation. And why not use USB for an optical drive? I'm seeing USB optical drives, and SATA drives in enclosures, somewhat regularly in Ottawa thrift stores for 5 to 10 dollars. More convenient anyway for sharing drives around the office, especially if any of you have laptops.</p>
<p>Modern ATX cases often don't have room for 4 full-size disk drives. You can probably find a way to use 5 or 6 SATA ports if you try, maybe 12 if you go for some "NAS" chassis like the DS380, but personally I'm using zero SATA ports in my newest system (after, like you, wanting a board with 6). Those external disks are what tend to be on sale here, and one of mine has a built-in hub. I've had zero bus-related problems, the performance is fine, and, again, it's just easier to deal with failures and upgrades. So I just never bothered shucking the drives; only my M.2 system drives are internal now.</p>
<p>I've never seen a board with 3 ethernet ports. But motherboards with 4 or 5 PCIe slots are easily found, and what is there apart from graphics and network cards to install in them? Built-in audio is ubiquitous and decent, as is built-in ethernet—one port but probably 2.5G—and wi-fi. So, a dual-ethernet card can occupy one; check the pricing for 2.5G or even 10G before you decide on gigabit. (Historically, some people with Bell fibre connections have liked to get cards with SFP ports in order to bypass the provided routers, but I hear some of the newer routers don't have removable modules.) I've also used USB ethernet dongles without trouble, but can't comment on performance or long-uptime reliability; very handy, though, when I have some piece of equipment that needs to be temporarily isolated from the internet.</p>
<p>Will you want ECC RAM? When I was buying, the Ryzen chips with onboard graphics didn't support ECC, so I went with an "ASUS ROG STRIX B550-E Gaming" board (despite not being much of a gamer) and an external RX 6600 graphics card. The other "catch" that might push you toward external graphics is that, last I checked, onboard graphics are generally limited to 2 outputs. The system I built works well, and I'd consider using successor components, if available, for my next PC. My only real complaints were the price, the defective 2.5 Gbe support, and the lack of an easy setting to turn off the network LEDs. (The "confidential" Foxville datasheet has since become available on LibGen, and I was able to take care of the LEDs using "ethtool --change-eeprom".)</p>
<p>If you're looking for more USB ports, be sure all the internal headers are populated—that's an extra 7 ports on my system, some on rear brackets—then maybe look to PCIe add-on cards (avoid anything with the vl805 chip; despite several attempts to update the firmware, I never got it working with Linux).</p>
</div>2024-02-09T19:24:38ZBy Chris Siebenmann on /blog/linux/MyMachineDesires2024tag:CSpace:blog/linux/MyMachineDesires2024:a6aadda9bdf688f2ccb0b44da12cfab4a45dcd99Chris Siebenmann<div class="wikitext"><p>My impression is that Wayland is most often based on some version
of OpenGL. I don't know how this interacts with Vulkan.</p>
</div>2024-02-09T15:31:45ZBy Ian Z aka nobrowser on /blog/linux/MyMachineDesires2024tag:CSpace:blog/linux/MyMachineDesires2024:2bffaeb203e5fd05c5d606a745b461712e149cbdIan Z aka nobrowser<div class="wikitext"><p>Is OpenGL still even a thing? I'm reading about Vulkan everywhere. (I hate it because of the name. Hate. Hate !!)</p>
</div>2024-02-08T22:47:23ZBy Miksa on /blog/linux/MyMachineDesires2024tag:CSpace:blog/linux/MyMachineDesires2024:52ce4a6689b5a25dcc4f0a5d46244ece9d84ec24Miksa<div class="wikitext"><p>Those specs are probably achievable with higher end motherboards. 6 SATA ports is also quite common, although using NVMe often disables couple. Larger NVMes are also quite within reach with steadily falling prices, this week Crucial had a sale of 7.6TB U.3 NVMe for $350.</p>
<p>For the 2.5G networking, beware that Intels 2.5G integrated offerings have given bad experience for a lot of people, even Realtek seems to be preferable.</p>
<p>For graphics needs it is unfortunate that inexpensive passive cooled and slot powered cards from modern generation are hard to come buy, mainly because all GPUs are too expensive.</p>
</div>2024-02-08T09:48:02ZBy Andrew on /blog/linux/MyMachineDesires2024tag:CSpace:blog/linux/MyMachineDesires2024:e82a537e2ad5ca5e188978ca44a85ceb8789ad68Andrew<div class="wikitext"><p>I think all of that is quite easy to come across except for the 3x M.2 (which does exist but it's not the most common because it means stealing PCIe lanes from somewhere else. And 6x SATA is getting less common than it used to be because of, well, M.2.</p>
<p>amdgpu has come along quite a bit in the past few years, due in part to Valve. My new laptop has a Radeon 780M integrated GPU, and it works rather smoothly.</p>
</div>2024-02-08T06:32:54ZBy Milo on /blog/sysadmin/TrackingMachineImportancetag:CSpace:blog/sysadmin/TrackingMachineImportance:a14cabb7a8465b1889cb79d4f522210199823b6bMilo<div class="wikitext"><p>Chapter 6 ("The Checklist Factory") of The Checklist Manifesto makes similar points: they shouldn't be static, they're definitely not free (to create or use, and thus can't cover everything), and testing is needed. If you're unfamiliar, it might be worth a trip to your excellent university library.</p>
<blockquote><p>There are good checklists and bad, Boorman explained. Bad checklists are vague and imprecise. They are too long; they are hard to use; they are impractical. They are made by desk jockeys with no awareness of the situations in which they are to be deployed. They treat the people using the tools as dumb and try to spell out every single step. They turn people’s brains off rather than turn them on.</p>
</blockquote>
<blockquote><p>Good checklists, on the other hand, are precise. They are efficient, to the point, and easy to use even in the most difficult situations. They do not try to spell out everything—a checklist cannot fly a plane. Instead, they provide reminders of only the most critical and important steps—the ones that even the highly skilled professionals using them could miss. Good checklists are, above all, practical.</p>
</blockquote>
<blockquote><p>The power of checklists is limited, Boorman emphasized. […]</p>
</blockquote>
<blockquote><p>[Testing] is not easy to do in surgery, I pointed out. Not in aviation, either, he countered. You can’t unlatch a cargo door in mid-flight and observe how a crew handles the consequences. But that’s why they have flight simulators, and he offered to show me one. […]</p>
</blockquote>
<blockquote><p>The three checklists took no time at all—maybe thirty seconds each—plus maybe a minute for the briefing. The brevity was no accident, Boorman said. People had spent hours watching pilots try out early versions in simulators, timing them, refining them, paring them down to their most efficient essentials.</p>
</blockquote>
<p>I doubt it's practical for you to spend weeks observing rookies in sysadmin simulators. But when I helped with on-boarding of new employees, I found it helpful to refer them to a "new employee" wiki page and ask them to edit it or ask questions as necessary; when they came to me with a question I thought should have been covered there, but wasn't, I could add it while we were speaking. Same for employees with more tenure: if you spent a bunch of time figuring something out, and it'll take less time to document it, write it on the wiki. (We also had a very bureaucratic "official document" process—so bureaucratic that, quite frankly, most of us didn't know what the process for revising a document was... so we didn't do that, and hence few people who weren't ISO auditors ever looked at them.)</p>
<p>As for the general idea of simulation: simulating a network of servers would take significant time to set up, and would never have 100% fidelity (there's always some dusty old machine in a closet that no remaining employee knows is important), but maybe should be something to aspire to. If done very well, (almost) the whole environment could be deployed into virtual machines for (limited) failure testing; if done extraordinarily well, with budgets unattainable to most sysadmins, testing could involve unplugging production machines randomly (cf. Netflix's "Chaos Monkey").</p>
</div>2024-02-07T21:04:09ZBy Chris Siebenmann on /blog/sysadmin/TrackingMachineImportancetag:CSpace:blog/sysadmin/TrackingMachineImportance:af8cb9fc0fd486650feb2def14ee1fb3850a2a5eChris Siebenmann<div class="wikitext"><p><a href="https://utcc.utoronto.ca/~cks/space/blog/sysadmin/UseAChecklist">I'm a big fan of checklists</a>, but at the same time
I think there are real issues. In general, <a href="https://utcc.utoronto.ca/~cks/space/blog/sysadmin/DocumentationIsNotFree">documentation isn't free</a> and checklists are a form of documentation.
For checklists related to failures, there's the additional issue that
<a href="https://utcc.utoronto.ca/~cks/space/blog/sysadmin/DocumentationNeedsTesting">documentation needs testing</a>, which can be
hard to do if you need an actual failure or a sufficiently accurately
simulated one to test your checklist with (and it also takes time).
In system administration, checklists generally can't be static things
that are created once, because the environment is constantly changing;
this means not just updating but re-checking and so on.</p>
<p>System environments are often sufficiently complicated that it's very
hard to foresee all effects of a failure or all interactions that your
systems have (some would say it's impossible). It's a classic story in
for the field that 'we thought we understood everything and had mitigated
everything, except surprise we hadn't'.</p>
<p>(Our checklists work best for routine things like installing machines
and for exceptional events that we can consider carefully in advance,
like planned power shutdowns.)</p>
</div>2024-02-07T19:10:32ZBy Milo on /blog/sysadmin/TrackingMachineImportancetag:CSpace:blog/sysadmin/TrackingMachineImportance:09dee5335b55e27127bdcf860b0800596cfdff47Milo<div class="wikitext"><p>In aviation, there are checklists for nearly everything. Regular maintenance, pre-boarding inspections, power-up, take-off, and of course the abnormal ones like "engine failure" and "rapid decompression". There have been some efforts to bring this mindset into other fields such as surgery. One of the major difficulties is trying to keep these short and helpful, rather than something that's accumulated "cruft" and is perceived as a clock-gobbling chore.</p>
<p>I wonder if that would be useful for system administrators. Like, a cooling failure checklist, post-power-outage checklist, DNS failure checklist, as so on. A big benefit in this field would be that many of the tasks, such as "new server bringup" and most things related to monitoring, could be scripted.</p>
</div>2024-02-07T16:26:17ZBy Miksa on /blog/sysadmin/TrackingMachineImportancetag:CSpace:blog/sysadmin/TrackingMachineImportance:cfb6903ee19324365d11d7479f09dcbf5522e168Miksa<div class="wikitext"><p>We had a similar experience few years back, but for a sillier reason. On a Sunday morning one of our datacenters started approaching 60C, bunch of servers had already turned themselves off preemptively and few of us showed to investigate and open doors for extra cooling. Sitting at the door we started pondering what could be the reason that 3 out 4 cooling units were turned off. We surmised that some kind of building automation controls the cooling based on temperature sensors and one way or another the data from the sensors needs to be transmitted out. One of us got a recollection or hunch that the cabinet with a door in the datacenter wall might have something to do with it and we decided to take a look.</p>
<p>Inside we find a small router-looking device and a small Eaton UPS. The device was off, an indication of a dead UPS, so we decided to try what happens if we unplugget the power cords from the UPS and connected them to each other. The device came alive and soon after the cooling units started turning back on.</p>
<p>The datacenter has an UPS the size of a large room, and it all comes crashing down because of a little UPS no one even remember existed.</p>
<p>This experience was a big intensive to go through the process you are considering. Our goal was to produce lists with startup order for the servers in case a datacenter had gone down. The first phase was documenting the role (dev/test/prod/administration) and priority tier (1-5) for all servers. We already had this information, but it was quite spotty. Annoying ordeal but not too bad in the end. Create lists, couple op staff go through them and add their opinions, then as a full group comb the list and negotiate an educated guess for all of them. Takes few hours but is worth it. A parallel task was to modify our ochestration tool to ask for this info for all new servers.</p>
<p>Next phase was to create script that creates list based on this information and the rack location of the servers. A great help for this was our scripted maintenance windows. Many frontend servers had scripts that make them wait until some other server has finished booting and some service, usually database, is back online. The backend server would automatically get at least .5 higher priority than the front. Then just a matter of uploading these lists to a website. Biggest remaining obstacle is regularly printing these lists to the datacenters.</p>
</div>2024-02-07T14:54:03ZBy Milo on /blog/sysadmin/TrackingMachineImportancetag:CSpace:blog/sysadmin/TrackingMachineImportance:0af4d566261fab89eba0be41beac2df7d5a629b0Milo<div class="wikitext"><p>We also don't want to end up like <a href="https://web.archive.org/web/20230610235249/http://bash.org/?5273">this person quoted on bash.org</a> (which seems to have gone offline recently):</p>
<p><erno> hm. I've lost a machine.. literally _lost_. it responds to ping, it works completely, I just can't figure out where in my apartment it is.</p>
</div>2024-02-06T17:45:16ZBy Jonathan Sharman on /blog/programming/GoAvoidingAnyAsATypetag:CSpace:blog/programming/GoAvoidingAnyAsAType:0cbccb68e501eed962ddf34aa4affeaae8e6b31cJonathan Sharman<div class="wikitext"><blockquote><p>Now we see clearly what Two() is doing differently than One(); it's obvious that it isn't taking 'any type' as such, but instead it's taking a generic interface as the argument type.</p>
</blockquote>
<p>This should already be obvious to someone familiar with generic programming. One function's parameter is of a generic type, and the other's isn't. And by your argument, shouldn't you prefer</p>
<pre>
func Two(v interface{String() string})
</pre>
<p>over</p>
<pre>
func Two(v fmt.Stringer)
</pre>
<p>In my opinion, the named interface is more readable and communicates intent more clearly in both cases, regardless of whether the interface is empty.</p>
<blockquote><p>it doesn't mean 'any type'; it means specifically the type 'an empty interface', which is to say an interface that has no methods</p>
</blockquote>
<p>Which is an interface value whose concrete type can be... <em>any type!</em></p>
</div>2024-02-06T16:17:55ZBy thefuga on /blog/programming/GoAvoidingAnyAsATypetag:CSpace:blog/programming/GoAvoidingAnyAsAType:c9032a2835efa6a8a962357852c7c3e91288ea68thefugahttps://github.com/thefuga<div class="wikitext"><p>To me the differences on both are very clear. They are, by no means, ambiguous. Readers should pay attention to what they're reading.
Of course, this is not to say clearity isn't important, but they're both clear. A programmer should never ignore that "[T any]" that's clearly screaming GENERIC.
"interface{}" was a bad choice. It is confusing to new comers and doesn't add any clearity. We can fix that mistake by using "any" from now on and stop insisting on a mistake. Why would that be a bad thing?</p>
</div>2024-02-06T16:01:19ZBy James on /blog/python/DjangoExplicitImportsSwitchtag:CSpace:blog/python/DjangoExplicitImportsSwitch:ba800f7b5bb6dc6707ac7c85a5741a22c604172eJames<div class="wikitext"><p>Explicit imports also let you know what you can mock for unit tests.</p>
<p>For example if you wanted to mock out the external call a python module makes, you would do:</p>
<p>```python</p>
<p>from unittest.mock import patch</p>
<p>def test_response():</p>
<pre>
with patch("myapp.module.fetch_google", return_value="somevalue") as m:
assert call_function_under_test() == "some_assertion"
</pre>
<p>```</p>
<p>And your test would return whatever needed value for some external call you wouldn't want to actually make.</p>
</div>2024-02-06T15:28:26ZBy Anonymous on /blog/sysadmin/TrackingMachineImportancetag:CSpace:blog/sysadmin/TrackingMachineImportance:f9fbc75c7279dd1bf9f4f78551a256124d7d5ac9Anonymous<div class="wikitext"><p>Perhaps (slightly) related: the first documented time people thought about this formally (as far as I can remember), was in this paper at Usenix/LISA '98: "Bootstrapping an Infrastructure" (<a href="https://www.usenix.org/legacy/event/lisa98/traugott.html">https://www.usenix.org/legacy/event/lisa98/traugott.html</a>) which might or might not still be relevant today.</p>
</div>2024-02-06T14:22:31ZBy goatops on /blog/sysadmin/TrackingMachineImportancetag:CSpace:blog/sysadmin/TrackingMachineImportance:ff78a76446d58c4850de362dc19ff113ad07edbagoatops<div class="wikitext"><p>We set up a simple color-coded sticker scheme for our ops team to follow in the event of a power failure where we wanted to maximise available power from the UPS. Each server was affixed with a green (can be shut down immediately), orange (can be shut down with notice) or red (try not to shut down at all) sticker on the front. Worked pretty well for us.</p>
</div>2024-02-06T12:25:48ZBy David Magda on /blog/sysadmin/TrackingMachineImportancetag:CSpace:blog/sysadmin/TrackingMachineImportance:ea8119e63bf30ae8e2c7cc7b79f4933fb8b08788David Magdahttp://www.magda.ca/<div class="wikitext"><p>If it is decided to try to document hardware, Netbox is a pretty good and worth looking at:</p>
<ul><li><a href="https://netbox.dev">https://netbox.dev</a></li>
<li><a href="https://docs.netbox.dev/en/stable/">https://docs.netbox.dev/en/stable/</a></li>
<li><a href="https://github.com/netbox-community/netbox">https://github.com/netbox-community/netbox</a></li>
<li><a href="https://demo.netbox.dev">https://demo.netbox.dev</a> (just click on "Log In": use any username/password, or click on "Sign In" for auto-generation)</li>
</ul>
</div>2024-02-06T12:02:29ZBy Arnaud Gomes on /blog/sysadmin/TrackingMachineImportancetag:CSpace:blog/sysadmin/TrackingMachineImportance:068721b1070c655d0bf70d55a4d21e73cdac8e6dArnaud Gomes<div class="wikitext"><p>We had the same kind of issues at a previous workplace, we ended up writing a "reboot the machine room HOWTO". It forced us to identify dependencies, and anything in the bottom half of the list was probably not essential.</p>
<pre>
-- A
</pre>
</div>2024-02-06T08:25:03ZBy Chris Siebenmann on /blog/programming/GoAvoidingAnyAsATypetag:CSpace:blog/programming/GoAvoidingAnyAsAType:a04e1947bbbde96ebb1c8cdb77d0d6003ec5f52cChris Siebenmann<div class="wikitext"><p>Whoops! I've fixed this now, and thank you for noticing and mentioning it.</p>
</div>2024-02-02T04:23:15ZBy anatol on /blog/programming/GoAvoidingAnyAsATypetag:CSpace:blog/programming/GoAvoidingAnyAsAType:575afd1372ee07e60db075ff1415a80965279b10anatol<div class="wikitext"><p>that 'is is why' can probably be reworded to something that puts less cognitive burden on the reader :)</p>
</div>2024-02-01T22:44:10ZBy djao on /blog/unix/WaylandTechnicalMeritsIrrelevanttag:CSpace:blog/unix/WaylandTechnicalMeritsIrrelevant:a9ec4b794c8a35655be355457567b6c9d3b9ee2adjao<div class="wikitext"><p>I'm in the second category: "People using mainstream desktop environments that already support Wayland." I've been using GNOME since 1999.</p>
<p>It's now 2024, and I've just started using Wayland full time, because</p>
<ol><li>My distribution (Ubuntu 24.04 prerelease) now defaults to Wayland, and</li>
<li>Wayland finally works well enough that I prefer it over what I had before.</li>
</ol>
<p>Firefox now defaults to Wayland and not XWayland, so that's a sign of progress. In fact, I've seen a LOT of progress in even just the last six months. For automation, ydotool is a fully capable replacement for xdotool. Screen sharing works well. But the argument is not just that Wayland can do what X11 could. (If that were all, then there would be no positive reason to switch.) In fact, Wayland is a noticeable improvement over X, and only after using it for some time does this improvement become apparent. Keep in mind, I am just a regular user, not a power developer, and I've always avoided Wayland because it didn't work. Well, now it works, and I see what all the fuss is about.</p>
<p>The biggest and most noticeable overall improvement is that the system is just more responsive in ways that matter to me. I lean heavily on focus-follows-mouse, and on X11, a focus transition into a window with a graphically complex titlebar (such as firefox with tabs in the titlebar) takes at least 200ms. On Wayland the same transition takes less than 50ms. How did I time this? By cycling the mouse pointer back and forth between windows and seeing how fast I can make the focus flicker. On X11, I can cycle at about 4Hz before the focus transition start to lag; on Wayland it's at least double that, and I'm not sure what the actual limit is because the system can go much faster than I can. Switching virtual desktops is likewise much more instantaneous on Wayland than X. By trilling two hotkeys (yes, I play the piano), I can continuously flip between two virtual desktops at about 10Hz on Wayland, and what is displayed on the screen is frame perfect the entire time. For all I know, Wayland can go faster, but I can't. The same experiment on X11 yields a maximum cycle rate of, again, ~4Hz, this time constrained by the system, and what is displayed on the screen during this time is an unrecognizable jumble of out-of-position windows from both desktops.</p>
<p>The point is not that I need to switch virtual desktops 10 times per second as part of my regular work. (I don't.) Rather, these examples are meant to state, in quantitative, measurable terms, why the Wayland user experience is better for regular users. Performance matters, not because it is strictly necessary (it's not; I managed to get work done on X for decades), but because a more responsive desktop is just a pleasure to use. In terms of work productivity, it's the same outcome, but I feel happier using Wayland.</p>
<p>It's also nice that Wayland fixes some mild annoyances. In X, you can't change the volume using keyboard shortcuts while the lockscreen is on! Wayland has no trouble with this. Better HiDPI support, and in particular better support for multiple monitors with different DPIs, is also a bonus.</p>
<p>Admittedly, I do not need specialized accessibility software. But, for my needs, Wayland is already better, and I'm not going back.</p>
</div>2024-01-30T09:48:56ZBy Peter Debelak on /blog/python/PythonVenvAndLSPServertag:CSpace:blog/python/PythonVenvAndLSPServer:94485e34d3c6c5159e643a716a86723e05c0db62Peter Debelak<div class="wikitext"><p>In case you haven't seen it, there's an emacs package called pyvenv that has a function pyvenv-activate to make emacs aware of your virtual environment. I have a single instance of pylsp installed via a virtualenv that is in my path and use it against various other virtual environments that I activate with pyvenv-activate while developing and everything works. It's all cpython, though, so I'm not sure if there would be a pypy vs. cpython issue for you.</p>
</div>2024-01-29T15:32:58ZBy -dsr- on /blog/sysadmin/ServersDroppingSerialPortstag:CSpace:blog/sysadmin/ServersDroppingSerialPorts:3b711944eb460c300394632a65afa96b0858b78d-dsr-https://blog.randomstring.org<div class="wikitext"><p>WTI ( <a href="https://www.wti.com/collections/console-servers">https://www.wti.com/collections/console-servers</a> ) makes reliable serial terminal servers. Fairly expensive, but also very long-lived.</p>
<p>Nevertheless, we've stopped putting them in datacenters in favor of IPMI.</p>
</div>2024-01-29T14:45:33ZFrom 193.219.181.219 on /blog/sysadmin/ServersDroppingSerialPortstag:CSpace:blog/sysadmin/ServersDroppingSerialPorts:a5dde821cf552bd0a9cdb80393c280417137a5f3From 193.219.181.219<div class="wikitext"><blockquote><p>The physical hardware for a serial port does add some cost and take up some space</p>
</blockquote>
<p>Routers and switches have long had a solution for that – RJ45 Cisco-style serial port. I'm surprised that servers never adopted it.</p>
<p>(Then again, I've just recently installed an iLO RJ45 addon module to one of our servers because the manufacturer included the whole BMC but for some reason decided that the dedicated Ethernet port should be something you buy as an addon...</p>
<p>On the other end of the spectrum, several of our switches have RJ45 <em>and micro-USB</em> for the serial console, with a PL2303 already inside the switch.)</p>
</div>2024-01-29T06:14:41ZBy Ian Z aka nobrowser on /blog/python/PythonVenvAndLSPServertag:CSpace:blog/python/PythonVenvAndLSPServer:a17bb352f13d6c3faba6eb5f72e6fa190b9f054dIan Z aka nobrowser<div class="wikitext"><p>I'd say that venvs themselves are "aesthetically displeasing". After all, having a separate Python executable for every project differs from having a separate LSP in degree only.</p>
</div>2024-01-28T17:48:21Z