First off, thanks to my friend Kirk Allen Evans for his help with this problem on a Saturday! Microsoft is lucky to have him. In the process he created a nice screen cast that you can watch to set the background. He doesn't encounter the bug, but as it is a threading issue he might have simply been lucky (or me unlucky threading bugs are like that), or (I'm hoping) he has a hotfix that I lack.
In some situations, a workflow will enter into a non-error, but unrecoverable state that looks as if an OnTaskCreated workflow activity did not fire even though the task exists. Sample code that reproduces the problem in my environment on both MOSS SP1 And MOSS SP1 + Infrastructure update is located here. If the problem occurs, the workflow will not continue processing beyond the OnTaskCreated activity. You can see reports of this issue as experienced by others here. Scroll down to the bottom of the post for a workaround.
I have a moderately complex workflow than starts three different tasks. A simplified version is shown below.
Intermittently, the workflow will stop working with history that makes it look like the workflow failed to handle one or more OnTaskCreated activities as shown below.
Notice that there is no entry for Task 1 Created (nor will there ever be). This workflow is toast and will not complete.
It turns out that the problem is actually in the workflow manager. If you step through the workflow, you will see every onTaskCreated activity light up in the designer, but the runtime will not invoke the method. If you look at the error log, you will see two exceptions:
RunWorkflow: System.ArgumentException: Item has already been added. Key in dictionary: 'c6923528-0908-4d75-86ed-f58342ea507e' Key being added: 'c6923528-0908-4d75-86ed-f58342ea507e' at System.Collections.Hashtable.Insert(Object key, Object nvalue, Boolean add) at System.Collections.Hashtable.Add(Object key, Object value) at System.Collections.Hashtable.SyncHashtable.Add(Object key, Object value) at Microsoft.SharePoint.Workflow.SPWorkflowManager.TrackCreatedInstance(Guid trackingId, SPWorkflow workflow) at Microsoft.SharePoint.Workflow.SPWorkflowManager.RunWorkflowElev(SPWorkflow originalWorkflow, SPWorkflow workflow, Collection`1 events, SPRunWorkflowOptions runOptions)
RunWorkflow: System.Workflow.Activities.EventDeliveryFailedException: Event "OnTaskCreated" on interface type "Microsoft.SharePoint.Workflow.ITaskService" for instance id "c6923528-0908-4d75-86ed-f58342ea507e" cannot be delivered. ---> System.NullReferenceException: Object reference not set to an instance of an object. at Microsoft.SharePoint.Workflow.SPWorkflowHostServiceBase.LoadInstanceData(Guid instanceId, Boolean& compressedData) at Microsoft.SharePoint.Workflow.SPWinOePersistenceService.LoadWorkflowInstanceState(Guid instanceId) at System.Workflow.Runtime.WorkflowRuntime.InitializeExecutor(Guid instanceId, CreationContext context, WorkflowExecutor executor, WorkflowInstance workflowInstance) at System.Workflow.Runtime.WorkflowRuntime.Load(Guid key, CreationCont...
Apparently, there is a threading issue with the Workflow manager and when it happens, the workflow receives notice that the event is firing, but it can't load the instance data.
I'm hoping that this is a known issue and that there is a hotfix available. I'll follow up as I learn more.
In the meantime, you can work around this problem by simply avoiding OnTaskCreated. The requirement I have that led me down this path is to set up permissions on the tasks beyond what the SpecialPermissions property allows (more on that some other day). I will meet that requirement by using a list event handler and apply the permissions outside the workflow via the handler.