I don't trust Linux distributions to leave directories alone
In yesterday's entry I said in passing:
My view is that basically every directory that your OS distribution creates is best left alone and unused, and thus should be left on the root filesystem. [...]
In theory there are any number of directories on typical Linux
distributions (and typical Unix distributions in general) that
should be safe for you to use without disturbance by the OS. There's
/opt, and yes, some of you are
laughing right now. In practice, I've been through enough experiences
that I no longer trust Linux distributions to leave any directories
they know about alone. Sooner or later someone is going to drop
files or subdirectories in there, or change the permissions or
SELinux context, or mandate that they must be on the root filesystem
because of some requirement, and so on and so forth. Sometimes the
guilty party will be the OS itself; sometimes it will be third
parties who are packaging things for the OS and decide that
/usr/local or whatever make a great place to put their stuff.
The practical reality of modern Linux life is that the only directories you can trust the OS not to screw with are directories that the OS has no idea exist, ie ones that you make up and create yourself. If the OS creates it, even if it's empty and explicitly marked 'for local sysadmin use only', using it is dangerous in practice. Sooner or later you're likely to regret it.
(Sometimes you have no choice because a program has been configured to look there or restrict itself to things there.)
Since directory names for local things are generally arbitrary anyways, you should make your life simpler and pick your own new names (I suggest organization-based ones).
The one exception to this is that if you package things in the distribution's native packaging scheme (.debs, RPMs, etc), my strong opinion is that you should default to putting them into the normal system locations even if it's local software. Sometimes this won't be possible (eg if you're packaging a conflicting version of a program), but when it is I think it's going to make your life easier. And as I've found out, there are things that really want to use the system locations.