Chris's Wiki :: blog/programming/COptimizerMakingProgramsCompile

Your C compiler's optimizer can make your bad programs compile

July 14, 2016

Every so often I learn something by having something brought to my awareness. Today's version is that one surprising side effect of optimization in C compilers can be to make your program compile. The story starts with John Regehr's tweets:

@johnregehr: tonight I am annoyed by: a program that both gcc and clang compile at -O2 and neither compiles at -O0

@johnregehr: easier to arrange than you might think

void bar(void);
static int x;
int main(void) { if (x) bar(); }

(With the Fedora 23 versions of gcc 5.3.1 and clang 3.7.0, this will fail to compile at -O0 but compile at -O1 or higher.)

I had to think about this for a bit before the penny dropped and I realized where the problem was and why it happens this way. John Regehr means that this is the whole program (there are no other files), so bar() is undefined. However, x is always 0; it's initialized to 0 by the rules of C, it's not visible outside this file since it's static, and there's nothing here that creates a path to change it. With optimization, the compiler can thus turn main() into:

int main(void) { if (0) bar(); }

This makes the call to bar() unreachable, so the compiler removes it. This leaves the program with no references to bar(), so it can be linked even though bar() is not defined anywhere. Without optimization, an attempt to call bar() remains in the compiled code (even though it will never be reached in execution) and then linking fails with an error about 'undefined reference to `bar''.

(The optimized code that both gcc and clang generate boil down to 'int main(void) { return 0; }', as you'd expect. You can explore the actual code through the very handy GCC/clang compiler explorer.)

As reported on Twitter by Jed Davis, this can apparently happen accidentally in real code (in this case Firefox, with a C++ variant).

Comments on this page:

By Aristotle Pagaltzis at 2016-07-14 04:36:33:

Well, to be pedantic, the program compiles either way. It’s the linker that errors out when given the unoptimised object.

This matters, because it means that the problem is in the abstraction of separating the compiler and the linker from each other. Given how the compiler’s and the linker’s jobs are defined, what happens here is totally kosher… even though undesirable. And so I cannot think of any way to fix it without piercing this encapsulation.

I guess the simplest way to fit this into the model would be for the compiler to output no-op linker symbols that need to be satisfied but not linked – i.e. by leaking responsibility from the compiler to the linker.

By skeeto at 2016-07-14 07:31:54:

I've run into a more sinister situation along these lines. My program was dynamically linking against a library libfoo and at runtime was using dlopen(3) to load a library libbar, which itself also linked against libfoo. Throughout the run of the program, it would occasionally dlclose libbar and then dlopen a newly compiled version.

However, with some versions of gcc and compiled with optimization, the program would crash in libfoo due to its internal state being corrupted. It took me awhile to figure it out: the optimizer, both in the compiler and linker, saw that the main program didn't actually make any calls into libfoo from reachable code and so didn't link libfoo at all. Every time the main program used dlclose on libbar, it was also unloading libfoo, and when libbar came up it wasn't reinitializing libfoo, assuming it would keep its global state.

Another note: You'll frequently get additional warnings compiling with optimization since the compiler is doing a more thorough analysis. I think that makes it worth compiling with optimization during development, only turning it off to examine the program from a debugger.

Written on 14 July 2016.

«	Our central web server, Apache, and slow downloads

Sudo and changes in security expectations (and user behaviors)