This Tech Tip was prompted first by our observation that OutputDebugString() didn't always work reliably when Admin and non-Admin users tried to work and play together (on Win2000, at least). We suspected permissions issues on some of the kernel objects involved, and in the process ran across enough information that we had to write it up.

We'll note that though we're using the term "debugger", it's not used in the Debugging API sense: there is no "single stepping" or "breakpoints" or "attach to process" going on like one might find in MS Visual C or some real interactive development environment. Any program that implements the protocol is a "debugger" in this sense. This could be a very simple command-line tool, or one more advanced such as DebugView from the very smart guys at SysInternals.

Table of contents

• Application program usage
• The Protocol
• The permission problem
• Detailed implementation
• Random thoughts
• Unixwiz.net tool: dbmutex

Application program usage

Simply calling OutputDebugString() with a NUL-terminated string buffer causes the message to appear on the debugger, if there is one. Common usage builds a message and sends it

---

sprintf(msgbuf, "Cannot open file %s [err=%ld]\n", fname, GetLastError());

OutputDebugString(msgbuf);

---

---

#include <stdio.h>
#include <stdarg.h>
#include <ctype.h>

void __cdecl odprintf(const char *format, ...)
{
char	buf[4096], *p = buf;
va_list	args;

        va_start(args, format);
        p += _vsnprintf(p, sizeof buf - 1, format, args);
        va_end(args);

        while ( p > buf  &&  isspace(p[-1]) )
                *--p = '\0';

        *p++ = '\r';
        *p++ = '\n';
        *p   = '\0';

        OutputDebugString(buf);
}

---

---

        ...
        odprintf("Cannot open file %s [err=%ld]", fname, GetLastError());
        ...

---

The protocol

object name	object type
DBWinMutex	Mutex
DBWIN_BUFFER	Section (shared memory)
DBWIN_BUFFER_READY	Event
DBWIN_DATA_READY	Event

The DBWIN_BUFFER, when present, is organized like this structure. The process ID shows where the message came from, and string data fills out the remainder of the 4k. By convention, a NUL byte is always included at the end of the message.

---

struct dbwin_buffer {
        DWORD   dwProcessId;
        char    data[4096-sizeof(DWORD)];
};

---

When OutputDebugString() is called by an application, it takes these steps. Note that a failure at any point abandons the whole thing and treats the debugging request as a no-op (the string isn't sent anywhere).

1. Open DBWinMutex and wait until we have exclusive access to it.
2. Map the DBWIN_BUFFER segment into memory: if it's not found, there is no debugger running so the entire request is ignored.
3. Open the DBWIN_BUFFER_READY and DBWIN_DATA_READY events. As with the shared memory segment, missing objects mean that no debugger is available.
4. Wait for the DBWIN_BUFFER_READY event to be signaled: this says that the memory buffer is no longer in use. Most of the time, this event will be signaled immediately when it's examined, but it won't wait longer than 10 seconds for the buffer to become ready (a timeout abandons the request).
5. Copy up to about 4kbytes of data to the memory buffer, and store the current process ID there as well. Always put a NUL byte at the end of the string.
6. Tell the debugger that the buffer is ready by setting the DBWIN_DATA_READY event. The debugger takes it from there.
7. Release the mutex
8. Close the Event and Section objects, though we keep the handle to the mutex around for later.

On the debugger front, it's a bit simpler. The mutex is not used at all, and if the events and/or shared memory objects already exist, we presume that some other debugger is already running. Only one debugger can be in the system at a time.

1. Create the shared memory segment and the two events. If we can't, exit.
2. Set the DBWIN_BUFFER_READY event so the applications know that the buffer is available.
3. Wait for the DBWIN_DATA_READY event to be signaled.
4. Extract the process ID NUL-terminated string from the memory buffer.
5. Go to step #2

The Permissions Problem

The mutex object is alive and allocated until the last program using it closes its handle, so it can remain long after the original application which created it has exited. Since this object is so widely shared, it must be given explicit permissions that allow anybody to use it. Indeed, the "default" permissions are almost never suitable, and this mistake accounted for the first bug we observed in NT 3.51 and NT 4.0.

The fix - at the time - was to create this mutex with a wide-open DACL that allowed anybody to access it, but it seems that in Win2000 these permissions have been tightened up. Superficially they look correct, as we see in this table:

SYSTEM	MUTEX_ALL_ACCESS
Administrators	MUTEX_ALL_ACCESS
Everybody	SYNCHRONIZE | READ_CONTROL | MUTEX_QUERY_STATE

The surprise occurs when one looks at the behavior of CreateMutex() when the object already exists. In that case, Win32 behaves as if we were calling:

OpenMutex(MUTEX_ALL_ACCESS, FALSE, "DBWinMutex");

Even deciding to perform all software development as an administrator is not a complete fix: if there are other users (services, for instance) that run as non-admins, their debugging information will be lost if the permissions are not right.

Our feeling is that the real fix requires that Microsoft add a parameter to CreateMutex() - the access mask to use for the implied OpenMutex() if the object already exists. Perhaps someday we'll see a CreateMutexEx(), but in the medium term we have to take another approach. Instead, we'll just hard-change the permissions on the object as it lives in memory.

This revolves around the SetKernelObjectSecurity() call, and this fragment shows how a program can open the mutex and install a new DACL. This DACL remains even after this program exits, as long as any other programs maintain HANDLEs to it.

...
// open the mutex that we're going to adjust
HANDLE hMutex = OpenMutex(MUTEX_ALL_ACCESS, FALSE, "DBWinMutex");

// create SECURITY_DESCRIPTOR with an explicit, empty DACL
// that allows full access to everybody

SECURITY_DESCRIPTOR     sd;
InitializeSecurityDescriptor(&sd, SECURITY_DESCRIPTOR_REVISION);
SetSecurityDescriptorDacl(
        &sd,            // addr of SD
        TRUE,           // TRUE=DACL present
        NULL,           // ... but it's empty (wide open)
        FALSE);         // DACL explicitly set, not defaulted

// plug in the new DACL
SetKernelObjectSecurity(hMutex, DACL_SECURITY_INFORMATION, &sd);
...

Our dbmutex tool performs just this job: it launches at system boot time, opens or creates the mutex, and then sets the object's security to allow wide access. It then sleeps until shutdown, holding the mutex open in the process. It consumes no CPU time.

Detailed implementation

We are purposely skipping most of the error checking: if things go wrong, it frees up allocated resources and exits as if no debugger were available. The goal here is to show the general behavior, not a complete reverse engineering of the code.

The "setup" function - whose name we have manufacturered - creates the mutex or opens it if not already there. They go to some pains to set the security on the mutex object so that anybody can use it, though in practice we'll see that they haven't quite gotten it right.

• OutputDebugString.txt

Random thoughts

But unlike most problems of this type, this is less intentional than most. Most mistakes are where the developer explicitly asks for too much (e.g., "MUTEX_ALL_ACCESS"), but this mask is implied by the behavior of CreateMutex(). This makes it harder to avoid without a change in the Win32 API.

---

While picking apart OutputDebugStringA() in KERNEL32.DLL, it became apparent how a non-admin could likely cripple a system. Once the mutex has been acquired, an appliation wishing to send a debug message waits up to ten seconds for the DBWIN_BUFFER_READY event to become ready, giving up if it times out. This seems like a prudent precaution to avoid starvation if the debugging system is busy.

But the earlier step, waiting for the mutex, has no such timeout. If any process on the system - including a non-privilged one - can open this mutex asking for SYNCHRONIZE rights, and just sit on it. Any other process attempting to acquire this mutex will be stopped dead in its tracks with no time limit.

Our investigation shows that all kinds of programs send random bits of debugging information (for instance, the MusicMatch Jukebox has a keyboard hook that's very chatty), and these threads are all halted by a few lines of code. It won't necessarily stop the whole program - there could be other threads - but in practice, developers don't plan on OutputDebugString() will be a denial-of-service avenue.

---

Oddly enough, we found that OutputDebugString() is not a native Unicode function. Most of the Win32 API has the "real" function to use Unicode (the "W" version), and they automatically convert from ASCII to UNICODE if the "A" version of the function is called.

But since OutputDebugString ultimately passes data to the debugger in the memory buffer strictly as ASCII, they have inverted the usual A/W pairing. This suggests that for sending a quick message to a debugger even in a Unicode program, it can be done by calling the "A" version directly:

OutputDebugStringA("Got here to place X");

Behavior under XP

Navigate: More Tech Tips

[Home] Stephen J. Friedl Software Consultant Tustin, California USA