50 Bytes of Code That Took 4 GB to Compile

While doing evil things with macros and the inline assembler (trying to run a weird test whose purpose is not really relevant) I managed to write a program that caused Visual Studio’s C++ compiler to allocate 4 GB of memory and then die.

Not bad for a program that can easily fit into a single 50 column line.

I might not have noticed except that my machine didn’t have 4 GB free at the time, and the frantic paging-out of data needed to find 4 GB of memory made my laptop completely unresponsive for a couple of minutes. If you have a machine with more than 4 GB free then this can be a good test case for doing memory analysis with ETW, to see if you can duplicate my results.

I’ve simplified the code down to its minimal essence because it amuses me:

void test()
{
    __asm { add eax
    __asm { add eax
}

Here is the compiler output:

error C2414: illegal number of operands
error C2414: illegal number of operands
error C2400: inline assembler syntax error in ‘opcode’; found ‘end of file’
fatal error C1060: compiler is out of heap space

I’m running 64-bit Windows and the compiler is a 32-bit large-address-aware process, so running out of heap space means allocating about 4 GB of address space. I did a few compiles in a row and you can see the 4GB spikes in memory usage from each one.

Then I got curious and wondered which part of the compiler was allocating all of this memory. I used etwheap.bat to record all heap and VirtualAlloc allocations from cl.exe and compiled the source file again. If I was doing it now I would use UIforETW – just make sure that VirtualAlloc stacks always is checked in the Settings dialog, or record a heap trace. There was just a couple of MB allocated from the heap, but lots allocated using VirtualAlloc, as shown here:

(etwheap.bat ships with UIforETW – run it from an elevated command prompt and follow the instructions – or just use UIforETW with VirtualAlloc call stacks, or UIforETW’s heap tracing for greater details)

And then, to finish off the investigation, I looked at the call stacks. We can see that the inline_assembler’s lexer is allocating a lot of Asm Tokens using its own VirtualHeap. VirtualHeap::Create reserves the address space, and VirtualHeap::HeapExtend commits the memory. Drilling down a bit deeper (not shown) shows that address space is reserved in 512 KB chunks, and is committed in 32 KB chunks.

There’s a few details that aren’t quite clear, such as why VirtualHeap::HeapExtend calls VirtualHeap::Create, but without source code that is an unknowable.

And so we stop. I’ll pass this along to the VC++ team as usual, and I wouldn’t be surprised if they fix it, but it’s not exactly a critical problem. I only noticed it because my machine didn’t have 4 GB free when I first hit this problem.

It’s a good thing the compiler is a 32-bit process or else it would have continued consuming memory beyond 4 GB. Three cheers for artificial limits!

These tests were done using VC++ 2010 on a debug build. I didn’t try any other variations.

Linux variation

A thematically similar problem (the linker consuming vast quantities of memory on a simple program is described on stackoverflow.

Windows Suxs?

I anticipate that some people will say that Windows is a horrible operating system and that is why my laptop locked up for a few minutes when this initially happened. Well, maybe, but if you allocate (and write to) 4 GB of RAM on your Linux and OSX machine and fail to cause a serious system slowdown that doesn’t actually prove anything. My laptop has 8 GB of RAM and most of it was in use, so the only possible way to get 4 GB of memory was to write a lot of data to disk. Laptop drives are notoriously slow. If I do the same test on my work machine (32 GB of RAM, 20 GB available) or my laptop when I have fewer programs running (5 GB free) then the 4 GB is allocated and freed in less than five seconds.

The reddit discussion can be found here.

Updates

It’s weird how some blog posts are more popular than others…

Some people had difficultly reproing this problem. The bug is only confirmed to happen in VS 2010 SP1, and it is only guaranteed to happen if the test function is the last thing in the source file.

This is clearly not a critical bug – the code is malformed, the compiler gives some warnings that point to the problem area, and nothing truly bad happens. However it is still a lexer failure. In particular, the out-of-memory failure prevents VC++ from reporting on the brace mismatch – if you add a function after test then the lexing completes and additional warnings are displayed.

Giving great error messages when compiling incorrect code is important enough that it is one of clang’s explicit design goals.