There's a reasonably popular view that having your firewall drop undesired packets instead of sending ICMP rejections is 'more secure'. It is not; instead it is 'more annoying'.

(Technically this is not quite true; in a very limited set of circumstances, dropping all packets for a host can hide some information from attackers. The flipside is that dropping some but not all packets usually leaks information about what you're screening.)

Dropping packets is more annoying because attempted connections have to time out before they fail, instead of failing immediately with some species of 'connection refused'. Note that this doesn't matter much to attackers, who are using automated software (which may have its own rapid timeouts).

When you use DROP instead of REJECT, you are being hostile to the other end. Hopefully you are being deliberately hostile, having determined that they are up to evil or are otherwise undesirable.

(For example, we use DROP for kernel SMTP screening because once someone makes it into our kernel level SMTP blocks we actively don't like them and want to cause them what annoyance we can. I accept that it's not likely to be very much annoyance.)

Using DROP on connection attempts from the inside to the outside is especially problematic. Unless you are overloaded with internal security problems, most of those connection attempts have a human behind them, one who your firewall is busy annoying. Hopefully not very many of the people on the inside of your firewall are evil and deserve to be annoyed.

Also, people have a much easier time diagnosing 'connection refused' than 'mysterious connection timeouts'. It can even stop them from retrying their connection several times (and cluttering up your logs or counters).

(See also Drop versus Reject for another discussion of this issue.)