Some reasons why CPUs might re-use unofficial NOPs for other things
A while back I wrote about an instruction oddity in the PowerPC 64-bit architecture, where the architecture reused a number of CPU instructions that otherwise had no effect in order to signal various things to the CPU. You might reasonably wonder why the PowerPC ISA decided to use unofficial NOPs for this, instead of using explicit dedicated instructions. Although I don't know the actual reasons, I can think of some reasonable explanations.
First, this reuse of instructions is fully compatible with CPUs
that don't support this feature, either because they're old or
because it's not applicable to them (for example, because they
don't support hardware threads). Of course, new CPUs that don't
care about new instructions could always ignore them, but that
would likely require some work to recognize the new instructions.
Here, a non-supporting CPU can simply handle the instruction as a
or' instruction that will wind up having no effect.
This is potentially especially useful if these instructions might
occur relatively commonly in 'hot' situations, where you don't want
to take various sorts of overhead by having to check CPU features
and so on.
(There are various approaches to this, but they all have some sort of overhead.)
Second, there may be only a limited amount of explicitly reserved instruction space in the instruction set architecture, and the ISA may have to last for decades as you keep expanding it to cover more and more features (consider all of the vectorization instructions that keep appearing in the 64-bit x86 ISA). You might not consider every potential feature to be valuable enough to permanently consume some of your remaining free instruction space. Reusing unofficial NOPs is an attractive proposition by contrast, since the instruction space is already allocated.
Third, using unofficial NOPs instead of new instructions also provides you a graceful off-ramp to no longer supporting the feature, if you decide it's not worth it any more. You can just remove the special handling of the instruction and let it return to its old meaning of just being an unofficial NOP (of which you may have a lot). People's code keeps running and you don't need any additional work in your future CPUs to recognize and ignore these instructions.
(This was sort of sparked by one of the comments here.)