Why does UnhookWindowsHookEx() fail?

I’m debugging a Windows hook DLL that’s misbehaving. Actually I’m still not sure whether it’s the hook DLL, or the C# library that calls it, that is the culprit. Here’s the problem symptom. My app DLL installs a mouse hook into a running application. The running app terminates. My app detects that it has terminated and attempts to remove the hook—this fails (1). On subsequent startup of the app in question, I try to re-install the hook onto it, and that fails (2). I’m actually more interested in failure #2 than failure #1, but the latter is probably a consequence of the former. So I needed to track down why #1 happened.

Side fact:

When you install a thread-specific Windows hook, your hook DLL is actually loaded in 2 places: one instance is loaded in your app's process space, and the other in the target app's.

At first, I thought that Windows was failing to do its DLL cleanup duties when the unhook step failed, so I was ready to look into manually forcing the unload and reload of the native DLL—often a bad idea to circumvent the garbage collector, but, hey, what else can you do?

Then I spotted the culprit:

DLLEXPORT HHOOK SetHook(HWND hWndTarget, HWND hWndListener) {
if(m_hWndListener != NULL) return NULL; // already hooked!
// install the hook via SetWindowsHookEx()
…
…
m_hWndListener = hWndListener;
…
}

DLLEXPORT BOOL ClearHook(HHOOK hook) {
BOOL unhooked = UnhookWindowsHookEx(hook);
if (unhooked) {
m_hWndListener = NULL;
// do other cleanup stuff
}
return unhooked;
}

It turned out: m_hWndListener was not being cleared when UnhookWindowsHookEx() failed.

Yet again, it goes to show that: just when I thought that the developers at Microsoft are a bunch of idiots who had no clue what they were doing when they invented the Windows API, it turns out that the idiot is me.

As to why UnhookWindowsHookEx() failed, my guess is that the hook was already being automatically removed by the OS when the target app terminated.