Tracking down compiler bugs

There was a time when it was "cool" to find a C compiler bug, but those days are long gone. This is especially the case with SCO compilers, which have had more bugs than ll the other development systems I've used put together.

This weekend I was building the wonderful rsync tool on a customer's OpenUNIX 8 system, but it core dumped early and often.

Many hours later I tracked it down to code that looked like this:

strcpy(alloca(n), str);

alloca is a compiler intrinsic that makes room on the stack for a variable of the given size ("n"), and it's de-allocated automatically when the enclosing function exits. This facility has been around a long time but has been somewhat problematic: this shows one of the reasons why.

The SCO compiler performed these steps (in simplified form)

• push "str" onto the stack
  
• allocate "n" bytes on the stack, which changes the stack pointer
  
• push the pointer to allocated memory
  
• call strcpy()
  

The strcpy() library function saw the pointer to allocated memory, but the original "str" parameter was waaaaay up higher on the stack, so it used as source data something other than the source string. It turns out that this stack-misalignment also had a cascading effect because saved registers were restored to the wrong place, and it just made the code go boom.

I believe that calling alloca() from within another function call is asking for trouble (even if legal), so the workaround in rsync is to use a temporary variable to hold the pointer to the new memory.

I've submitted a bug report to SCO and a patch to the rsync folks.

Posted by Steve at December 23, 2002 02:24 PM | TrackBack