The security danger of exploitable bugs in file format libraries
Lately there have been a raft of security bugs of the form 'the standard open source library for dealing with file format X can be made to do bad things if it opens a specially crafted file in that format'. Some of the time that is 'run arbitrary code'. A bunch of these bugs have been found with the new fuzzer afl; see its 'bug-o-rama' trophy case.
At one level, that these bugs exist in many libraries for handling various file formats is not too surprising. A great deal of them are old libraries, and in general such libraries have generally assumed that they are not being run in any security sensitive context; instead they usually assumed you were using them on your own files and if you want to run arbitrary code as yourself, well, you already can. This can lead to these bugs not seeming too alarming.
There are two reasons to be worried about this today. First, in practice you get a lot of files from other people over the Internet; your browser downloads them (and often tries to display them), your mail client gets them in mail (and often tries to display them), and so on. However this is mostly a desktop risk and is relatively well understood (and many browsers and mail clients are using hardened libraries, although people keep finding new attack points).
Unfortunately there is another risk on Unix systems, and that is smart services that attempt to do content type detection and then content conversion for you. The dangerous poster child for this is the CUPS printer system, but there are probably others out there. In normal default setups, CUPS will try very hard to take random files that users hand it and turn them into something printable. This process involves both questionable content sniffing and, obviously, reading and interpreting all sorts of file formats. CUPS almost certainly uses standard libraries and programs for all of this, which means that exploitable vulnerabilities in these libraries can be used to break into the CUPS user on any system where CUPS is doing these conversions (and CUPS likes doing them on the print server).
(Another possible attack vector is email anti-spam, anti-virus systems. These almost certainly open .zip files using some library and may try to do things like peer inside PDFs and various '* Office' file formats to look for bad things.)
In general we've had a whole parade of troubles with any system that reads attacker-supplied input. We really should be viewing such things with deep suspicion and limiting their deployment, even if it's too late in the case of CUPS.