2021-05-24
Rust is a wave of the future
Recently I tweeted:
Once again I am being tempted to try out Rust, despite a relatively high confidence that I'm going to not like it. It's the wave of the future, though. Sooner or later I'm going to have to read and hack on Rust code.
More than five years ago, I wrote an entry on my feelings on using on Rust myself which I have since summarized to people as 'Rust cares about things that I don't any more'. I understand that Rust has much better ergonomics than it did in 2015 (when Rust 1.0 was just out), and so some of my other issues might be better.
But that's not why Rust is a wave of the future (in the manner of tweets, I said 'the wave'). Rust is a wave of the future because a lot of people are fond of it and they are writing more and more things in Rust, and some of these things are things that matter to plenty of people. There's Rust in your Python cryptography. There's Rust in Curl (sort of) (also). There's Rust in your librsvg. There's a lot of Rust in your Firefox. There are a growing number of command line tools written in Rust, including the excellent ripgrep. Someday there will probably be Rust in the Linux kernel. All of this is only growing with time, especially in the open source world.
All of this means that dealing with Rust is going to become increasingly necessary for me and a lot of people. We may not write it, but we need to be able to deal with programs that use it and are written in it. Our systems increasingly need a Rust build environment, I recently explored Rustup, and sooner or later I'm going to have to read and perhaps change Rust code in order to understand some problem we're having and perhaps fix it. Learning how to write some Rust myself is one way of getting the experience and knowledge necessary to do that well.
By the way, this isn't some nefarious conspiracy to force Rust on everyone. Some of it is that Rust really does solve problems that people have, but as I mentioned, a larger part of it is that lots of people genuinely like writing in Rust. When people like a programming language, things get written in that language and some of them become widely used or popular (or both). We saw this with Go in certain areas, and we're seeing this with Rust.
(There's an interesting question, one that I don't know the answer to, about why we haven't seen this for various other languages. Of course we have seen this for some; Javascript is large now and there are general tools written in it, and Java was large once upon a time. Some of it is likely that Rust can be airlifted into C and C++ programs relatively easily, but that doesn't explain programs written from scratch.)
Why we don't have management connections to our switches (an old story)
One of the unusual things about our physical network implementation is that while we have managed switches, we don't have management connections to them, at least over the network (we do often have serial console connections to them). Since we don't have a network management connection, we can't use SNMP to pull data from our switches, which is somewhat inconvenient now that we have a Prometheus and Grafana setup; it would be nice to know things like per-port bandwidth and so on. While there are additional reasons to avoid networked management connections for some switches, our not using management connections has a lot to do with a bad experience with them years ago. This is the story of that bad experience.
(On our expensive 10G switches, we don't have the spare ports to dedicate one to a management connection.)
Our bad experience happened back in the era of our second network implementation, where many of our then new 24-port switches carried multiple VLANs and split them out on a per-port basis. These switches were interconnected in a tree for regular VLANs, and their management port was connected to an entirely physically separate management network that ran over its own simple non-VLAN'd, single network switch fabric using dedicated inter-building links. This setup meant that we had multiple paths between switches, but they were on different VLANs and so not subject to cross-talk or packet loops (there was one path through the regular VLAN'd tree, and one path through the management network).
One day we noticed that packets from one subnet were showing up on another subnet. This happens from time to time and is usually caused by someone connecting two drops together when they shouldn't be, for example by plugging both into a little desktop switch. What was weird about this case was that it wasn't all packets from the subnet, it was only occasional packets. When we traced through all of the switch and port fabric, we couldn't find any crossed-over drops or ports. Even unplugging bits of the network didn't stop these occasional packets showing up where they shouldn't be, but it did establish that they behaved very weirdly; for instance, packets generated in one building would prove to be injected on the other subnet in a completely different building. Eventually we noticed that these packets were appearing on the management network, in addition to their two regular subnets, which gave us the necessary clue.
What was really happening was that our 24-port switches seemed to have some sort of flaw when handling their management network. If the management network was connected up, sometimes a switch would decide to take a regular untagged packet on another VLAN and inject it into the management network, and also sometimes take an untagged packet from the management network and inject it into some other VLAN. Since our management network had no regular traffic of its own, almost all of the traffic available to be switched over to other VLANs was traffic that had started out on other VLANs and had been injected into the management network. The combination of both happening to the same packet would cause a packet to apparently teleport from one place on one network in one building to a completely different place on another network in another building.
Our solution was to remove all of our switches from our management network. For the remainder of their life as multi-VLAN switches (until we shifted over to our third network implementation), they were configured either through their serial port or by connecting a laptop directly to their management port.
Ever since then, we've never connected another switch's management port to our management network. Maybe it would work these days, or maybe there are other problems lurking, waiting to be discovered. So far, we don't need to talk to our switches remotely over the network badly enough to risk it.
(For the multi-VLAN switches at the core of our networks, we do talk to their serial consoles remotely through our serial console infrastructure.)