Debugging hybrid VB6/C/C# app crashes with WinDbg

I was troubleshooting one of two (2) different app crashes this past week in which the app simply dies without any prior warning.  At first I thought: crappy VB6! But then, after looking in the Event Log, I found out this was not Visual Basic 6.0 Runtime’s fault, but rather it’s the .NET Runtime version 2.0 Fatal Execution Engine that erred fatally. Below is how I went about debugging the crash.

First of all, two sets of tools are needed:

1. Debugging Tools for Windows (WinDbg)
2. Debugging symbol (.pdb) files. These are useful but not essential.   PDB files are generated automatically in Visual Studio .NET, and can be turned on in VB6 IDE via the project properties under the Compile tab.

In using WinDbg to troubleshoot app crashes, I found this blog post Using WinDbg - Hunting Exceptions  by Johan Straarup to be tremendously helpful.

As anyone who’s ever involved in troubleshooting any software bug would know, the hardest part is to reproduce it in a consistent manner, and an app crash is (or should be) the rarest (and most feared by programmers) of bugs. Yet, an app crash is nothing more than an exception that has gone uncaught by the application.

In WinDbg, I used the Attach to a Process menu option to connect to the running app. Then, I issued the gn command (Go [until] Unhandled Exception) command, and let the app run its course.  Eventually, it came a stop with this message:

(4a4.1e4): CLR exception - code e0434f4d (!!! second chance !!!)
eax=0129ecfc ebx=0017d328 ecx=00000000 edx=00000025 esi=0129ed88 edi=e0434f4d
eip=7c812afb esp=0129ecf8 ebp=0129ed4c iopl=0         nv up ei pl nz na po nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00000202
…

I then used “!analyze –v” and subsequently “!pe”, which gave me this:

0:002> !PrintException
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\mscorwks.dll -
PDB symbol for mscorwks.dll not loaded
Exception object: 019138a0
Exception type: System.IO.FileLoadException
Message: Could not load file or assembly 'Interop.SHDocVw, Version=1.1.0.0, Culture=neutral, PublicKeyToken=null' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
InnerException: <none>
StackTrace (generated):
    SP       IP       Function
    00000000 00000001 MyAssembly1.MyClass1.Finalize()

So, this crash was the result of a referenced assembly version mismatch in one of the plugin components. That's one easy problem solved.

The other crash is a bit more elusive, which, after two weeks, I'm still in the process of trying to reproduce.

Why does UnhookWindowsHookEx() fail?

I’m debugging a Windows hook DLL that’s misbehaving. Actually I’m still not sure whether it’s the hook DLL, or the C# library that calls it, that is the culprit. Here’s the problem symptom. My app DLL installs a mouse hook into a running application. The running app terminates. My app detects that it has terminated and attempts to remove the hook—this fails (1). On subsequent startup of the app in question, I try to re-install the hook onto it, and that fails (2). I’m actually more interested in failure #2 than failure #1, but the latter is probably a consequence of the former. So I needed to track down why #1 happened.

Side fact:

When you install a thread-specific Windows hook, your hook DLL is actually loaded in 2 places: one instance is loaded in your app's process space, and the other in the target app's.

At first, I thought that Windows was failing to do its DLL cleanup duties when the unhook step failed, so I was ready to look into manually forcing the unload and reload of the native DLL—often a bad idea to circumvent the garbage collector, but, hey, what else can you do?

Then I spotted the culprit:

DLLEXPORT HHOOK SetHook(HWND hWndTarget, HWND hWndListener) {
if(m_hWndListener != NULL) return NULL; // already hooked!
// install the hook via SetWindowsHookEx()
…
…
m_hWndListener = hWndListener;
…
}

DLLEXPORT BOOL ClearHook(HHOOK hook) {
BOOL unhooked = UnhookWindowsHookEx(hook);
if (unhooked) {
m_hWndListener = NULL;
// do other cleanup stuff
}
return unhooked;
}

It turned out: m_hWndListener was not being cleared when UnhookWindowsHookEx() failed.

Yet again, it goes to show that: just when I thought that the developers at Microsoft are a bunch of idiots who had no clue what they were doing when they invented the Windows API, it turns out that the idiot is me.

As to why UnhookWindowsHookEx() failed, my guess is that the hook was already being automatically removed by the OS when the target app terminated.

stdole 7.0.3300.0

When I first got this error during the initialization phase of my component:

Could not load file or assembly 'stdole, Version=7.0.3300.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.

I thought that it wanted Interop.stdole.dll, which was sitting right there in the working directory. But as it turned out, it meant the Primary Interop Assembly called stdole.dll, which was indeed missing.  Duh!

Unable to cast COM object of type ‘X’ to interface type ‘_X’

I added a new property to my COM class and consequently bumped up its minor version by 1, did some testing with it, and then decided that I wanted to run a clean build.  Now my existing NUnit test cases that use this class are failing with the following message:

Unable to cast COM object of type MyLib.MyClass' to interface type MyLib._MyClass'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{…}' failed due to the following error: No such interface supported (Exception from HRESULT: 0x80004002 (E_NOINTERFACE)).,

The interesting thing is that everything compiled without a hitch.  Is this  .NET’s version of BadImplementsRefInCompatLib?

Looking that the COM library’s IDL info via oleview.exe , it turned out that the interop assembly was bound to version 1.3 of the COM library, whereas 1.2 was registered on my machine. I regenerated the COM interop assembly for 1.2, and the problem went away.

VB6 BadImplementsRefInCompatLib and .NET-COM Interop

You've created a VB6 COM library, MyLib1 (1.0).
You then have two usage scenarios for MyLib1:

1. MyLib1 is, in turn, used by another VB6 library, MyLib2


2. MyLib1 is used to generate a COM Interop assembly which is consumed by a .NET assembly called MyDotNetLib, which itself is also a COM interop library.

It so happened that you needed to add some new public methods to MyLib1, which would bring its versioning to 1.1. You were careful to craft the change such that it's backward compatible. Yet, something happened--you don't yet know exactly what it was--and when to try to recompile the above two libraries, you get the following problem scenarios:

1. Recompile MyLib2, you get this error: BadImplementsRefInCompatLib


2. Recompile MyDotNetLib, you get:
   Error 63 The assembly "..MyDotNetLib.dll" could not be converted to a type library. Type library exporter encountered an error while processing 'MyDotNetLib.MyComponent, MyDotNetLib'. Error: Referenced type is defined in managed component, which is imported from a type library that could not be loaded (type: 'MyLib1._IComponent'; component: '...\Interop.MyLib1.dll')
   

What did you do wrong?
As it turns out, the "something" in "something happened" above was an IDE debug session that had previously crashed, and MyLib1.vbp (1.1) remained registered in the system.

Why did that cause the above problems to occur?
The IDE looks for the latest version of the library (same GUID) to bind to.

How to avoid it?
Well, you can't avoid it, any more than you can avoid making VB6 crash.

What to do when it happens?

• Temporarily remove reference to MyLib1 from your MyLib2 project, then attempt to re-add the reference, looking through the list of registered libraries to see how many instances of MyLib1 are registered--there should be only one. If you see a MyLib1.vbp showing in your list, you'll need to go through the registry and manually remove all CLSIDs with InprocServer32="MyLib1.vbp". The automatic registry clean tools, that I've tried, couldn't remove these entries for me.


• Retrace your changes to make sure that you haven't made any modification to public interfaces that could break binary compatibility.


• Use the OLE/COM Object Viewer tool to view Type Library info of MyLib1.dll. Then compare the output with a previous version of the binary to see if any of the interface GUIDs got inadvertently changed.
  

---
Pre-emptive comment:
Yes, I know VB6 (the IDE) has reached its end-of-life, which means MS support for it will be scarce--all the more reason for writing this post.

NUnitForms rocks!

This NunitForms tool is really cool. Kudos to Luke Maxon and co. I'm running v2.0 alpha 5 and so far, my favourite feature has got to be ControlTester.FireEvent(). For instance, I can simulate the drag operation with something like this:

// initialize control tester for our Finder control
ControlTester myControl = new ControlTester("picFinder", "MyTestForm");

// simulate drag to coordinate (105,205)
myControl.FireEvent("MouseDown", (EventArgs)null);
Cursor.Position = new Point(105, 205); // drag mouse to here
myControl.FireEvent("MouseMove", (EventArgs)null);
myControl.FireEvent("MouseUp", (EventArgs)null);

// Assert MyTestForm GUI states
...

Pretty neat.

Rhino.Mocks ExpectationViolationException

I'm trying out Rhino.Mocks in my NUnit test suites. Why Rhino.Mocks instead of NMock, you ask? Well, because I've heard that Rhino.Mocks can mock classes as well as interfaces.

Anyway, I was running into a little bit of a rookie's setback. I set up my mock objects like this:

1   const string TESTNAME = @"My Component Name";
2   MockRepository mocks = new MockRepository();
3   MyLib.MyTestClass tw = mocks.CreateMock<MyLib.MyTestClass>();
4   Expect.Call(tw.get_Name()).Return(TESTNAME);
5   mocks.ReplayAll();
    // Test first invocation of get_Name()
6   Assert.AreEqual(tw.get_Name(), TESTNAME);
    // Test second invocation of get_Name()
7   Assert.AreEqual(tw.get_Name(), TESTNAME);

I should probably add that MyLib.MyTestClass is a COM Interop class.

Executing line 7 resulted in this exception: Rhino.Mocks.Exceptions.ExpectationViolationException : _MyTestClass.get_Name(); Expected #1, Actual #2.

A little bit of googling did not immediately hint at the proper solution for me, but looking at the API documentation turned up this method: Repeat.AtLeastOnce(). So adjusting line 4 above to this did the trick:

4   Expect.Call(tw.get_Name()).Return(TESTNAME).Repeat.AtLeastOnce();

Why it's good to use [System] Hungarian notation

One of our components has a property called--for reason of obscurity say--URL. The code behind it is quite simple:

        public string URL {
            get {
                return mstrURL;
            }
            set {
                mstrURL = value;
            }
        }

This property is being edited through our Property Editor component via a sort of reflection mechanism.

Sometime ago, this component broke because the Property Editor was no longer able to "reflect" on the URL property.

A bit of googling turned up this MSKB article: "Type library identifiers are not case sensitive by design".

It turned out that somebody recently added this method in a completely different class, within the same assembly, but entirely unrelated to the component above:

   public void LoadUrl(string url) {
      ...
   }

So the code that gets called by RegAsm to generate the type library must have found the url parameter of the LoadUrl method first, and decided to reuse it as the moniker for our URL property.

I guess the moral of this story is: Stick with System Hungarian Notation.

   public void LoadUrl(string strUrl) {
      ...
   }

(Side Note: A good article on Hungarian Notation can be found at Joel On Software).

Hosting .NET controls in VB6

I recently ran into a need to interop a C# .NET user control with a VB6 form.

We all know that VB6 only allows you to reference a user control if it resides in an OCX.

The Interop Forms Toolkit, mentioned here, seems to be only good for VB.NET user controls.

Prior to this, I've been able to get by with dynamically adding the control via Controls.Add {COMClassname}, {ControlName}

But now, I need to be able to manipulate the control in design time.

So, based on the above experience, I created a very basic VB6 user control called DotNetControlWrapper. The tricks with this control are the following:

• There's a public property called DotNetClassName, which is a COM class name of the .NET control.


• It delegates resizing logic to the .NET control.


• It exposes the .NET control via a DotNetObject property, thus allowing the user to sync the control's events if needed.

The bulk of DotNetControlWrapper.ctl looks like this:

Option Explicit

Private object As Object
Private ctlExtender As VBControlExtender
Private  mstrDotNetClass As String 

Public Property Get  DotNetClassName() As String
   DotNetClassName = mstrDotNetClass
End Property 
Public Property Let DotNetClassName(ByVal clsname As String)
   mstrDotNetClass = clsname
   If mstrDotNetClass = "" Then Exit Property
   ' initialize the .NET control
   If Not ctlExtender Is Nothing Then
     Controls.Remove UserControl.Extender.Name + "_dotnet"
   End If
   Set ctlExtender = Controls.Add(mstrDotNetClass, UserControl.Extender.Name + "_dotnet")
   ctlExtender.ZOrder 0
   Set object = ctlExtender.object
End Property

Private Sub UserControl_Resize()
   If object Is Nothing Then Exit Sub
   ctlExtender.Move 0, 0, UserControl.Width, UserControl.Height
   object.BringToFront ' Bring the .NET control to foreground
End Sub

Private Sub UserControl_Show()
   If ctlExtender Is Nothing Then Exit Sub
   ctlExtender.Visible = True
End Sub

Public Property Get DotNetObject() As Object
   Set DotNetObject = object
End Property

Private Sub UserControl_Terminate()
   If ctlExtender Is Nothing Then Exit Sub
   Set ctlExtender = Nothing
   object.Dispose
   Set object = Nothing
End Sub

Private Sub UserControl_WriteProperties(PropBag As PropertyBag)
   PropBag.WriteProperty "DotNetClassName", mstrDotNetClass
   PropBag.WriteProperty "Width", UserControl.Width
   PropBag.WriteProperty "Height", UserControl.Height
End Sub

Private Sub UserControl_ReadProperties(PropBag As PropertyBag)
   On Error Resume Next
   DotNetClassName = PropBag.ReadProperty("DotNetClassName")
   Width = PropBag.ReadProperty("Width")
   Height = PropBag.ReadProperty("Height")
End Sub