The Linux kernel's general IP environment has a system for marking packets with what is generally called a fwmark, short for 'firewall mark'. Fwmarks can be set through iptables, using the MARK target (documented in iptables-extensions), or by facilities such as WireGuard, and can then be used by firewall rules or by 'ip rule' policy based routing. Fwmarks are how I solved the general recursive routing problem when I set up my WireGuard environment. All of my uses of fwmarks have been simply picking a value, setting it, and checking for it. I was recently working with something that also uses fwmarks, and I saw unusual things in 'ip rules' and 'iptables' output:

# ip rule list
[...]
5210:   from all fwmark 0x80000/0xff0000 lookup main
[...]
# iptables -nL
[...]
MARK   all -- 0.0.0.0/0 0.0.0.0/0 MARK xset 0x40000/0xff0000
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 mark match 0x40000/0xff0000
[...]

(This is actually not all of the rule because I didn't use -v; the full MARK rule has an interface limitation.)

The thing after the '/' in this new to me syntax is a fwmark mask, which is sometimes called a fwmask. As suggested by its name, a fwmark mask restricts what portion of the fwmark you're matching or setting. A plain fwmark rule like 'from all fwmark 0x5151 ....' matches any packet with a fwmark that is exactly 0x5151 and nothing else; a rule with a fwmask matches if the portion of the fwmark selected by the mask matches. So the fwmask here means that any fwmark of the form '0x08xxxx' will match, regardless of lower order parts; a fwmark of '0x085151' would match just as well as '0x080000'. Similarly, setting a mark with a mask only affects the masked portion of the fwmark, not all of it. If I just set a 0x5151 fwmark on a packet, I overwrite any existing fwmark it had; if I use a mask, I could turn a packet with fwmark 0x080000 into one with fwmark 0x085151.

As suggested by this description, fwmark masks are aimed at situations where multiple pieces of your IP handling may all be trying to use fwmarks for their own purposes. Since a packet can only have one fwmark, you need some way to effectively combine multiple marks together. That's what fwmasks let you do, by having each different piece only use its own part of the fwmark. Coordinating who gets which part of the fwmark is up to you, although some piece of software may have just decided that it will take 0xff0000 and hope that this doesn't collide with other things. This is actually not a terrible approach and since there's no registry for fwmark usage it's hard to do better.

If you're curious (I was), the 'MARK xset' is the way 'iptables -L' prints what is created with '-j MARK --set-xmark 0x40000/0xff0000'. I assume that this particular piece of software sometimes sends in packets that are already set to a fwmark with 0x40000 turned on, in which case the --set-xmark will flip it off instead of turning it on.

(This handily illustrates that if you have a piece of software that uses fwmarks, you can't necessarily tinker with its usage or change its rules because you don't necessarily know what it's up to. This software reserves a full byte for its fwmask (0xff), while only apparently using two bits of it between 'ip rules' and iptables. Does it sometimes set and use the other bits? Maybe. In this case the program in question is open source so I can read the code if I need to. People interested in this particular case are directed here.)