Environment variables make bad switches
Someday you may be faced with a deprecation situation where you need to add a compatibility switch to your program. When in this situation it may be tempting to resort to controlling things with an environment variable, instead of giving your program yet another command line option.
(The problem with using command line options is that they add clutter, complexity, or both to your documentation, including any usage stuff that the program prints.)
Unfortunately this is not a good way to add such switches to your program, especially compatibility switches. Beyond cluttering up the environment in general, environment variables tend to leak, winding up in the environment of everything that a script runs. If any of those other programs and scripts also use your program, they're suddenly running it with different options than they expect; this may not work too well, to put it one way.
(Yes, in theory people can set the environment variable just when they run your program directly. But in practice they won't because it's too annoying; instead they'll just set the environment variable once at the start of their script and it will propagate.)
Or in less words: using environment variables makes it possible for one script to accidentally poison a different script's use of your program. Spooky action at a distance is not an asset when writing shell scripts.