Tracking down a subtle Win32 bug

On and off for several years, I've run into all kinds of problems with the Win32 function OutputDebugString(), which lets an application send a printable string to a debugger for display. This is a great way for "inaccessible" software - such as those running in a system service - to send debugging messages.

But there has been a long-running, sporadic problem (going back to NT 3.51) where these messages simply get lost, and now that I'm doing my software development under Windows as a non-admin user, it got worse, and it was crippling me on a project. Debugging is hard enough with debugging messages. So I figured it out, and I consider this to be a bug in Windows.

The results of several days of work are two things:

Tech Tip: Understanding Win32 "OutputDebugString"

This gives the background on how OutputDebugString() works, how to use it, and the subtle permissions issues that has been so troubling for so long. It includes a pseudocode for the function reverse engineered from KERNEL32.DLL.

Tool: dbmutex

This is a very small Win32 system service that launches at system boot time, fixes the permissions problem, and goes to sleep until shutdown. During this process it's holding onto a HANDLE to the troublesome Mutex object so the permissions are right every time.

This shouldn't have taken this much work, but at least it's now mainly behind me - now back to real work.

Posted by Steve at December 10, 2003 11:21 PM | TrackBack