Very Sleepy SharePoint Workflow Pauses

Ahh, the dreaded sleepy 5 minute SharePoint workflow action pause that takes days even weeks to complete. Classic symptom of this problem in the workflow history logs reveals a quite unhelpfully  message such as 

[Workflow Name]failed to run

The one helpful thing the error in the workflow history logs does give is the time of the error. Using this error time I would suggest to go though the SharePoint event logs of each server and find out which server servers that this error occurs on. In my case this was occurring only on a front end server that we had set up to to only serve search index requests. it appeared that every workflow that tried to run on this server in OWSTimer was basically spewing out a error message then  putting the putting the workflow back to sleep. 

The errors it spews out are:

Workflow Compile Failed while loading AuthorizedTypes from config: Configuration system failed to initialize

and 

RunWorkflow: Microsoft.SharePoint.SPException: Configuration system failed to initialize     at Microsoft.SharePoint.Workflow.SPNoCodeXomlCompiler.SubCompiler.DoCompile(WorkflowCompilerParameters parameters, String xomlSource, String assemblyName, CompilationPacket& packet, DirectoryInfo& tempDir)     at Microsoft.SharePoint.Workflow.SPNoCodeXomlCompiler.SubCompiler.DoCompile(WorkflowCompilerParameters parameters, String xomlSource, String assemblyName, CompilationPacket& packet, DirectoryInfo& tempDir)     at Microsoft.SharePoint.Workflow.SPNoCodeXomlCompiler.<>c__DisplayClasse.b__c()     at Microsoft.SharePoint.Utilities.SecurityContext.RunAsProcess(CodeToRunElevated secureCode)     at Microsoft.SharePoint.Workflow.SPNoCodeXomlCompiler.DoCompileNewAppDomain(WorkflowCompilerParameters parameters, String xomlSource, String assemblyName, SPWeb web, CompilationPacket& packet, DirectoryInfo& tempDir)     at Microsoft.SharePoint.Workflow.SPNoCodeXomlCompiler.CompileBytes(Byte[] xomlBytes, Byte[] rulesBytes, Boolean doTestCompilation, String assemblyName, SPWeb web, Boolean forceNewAppDomain)     at Microsoft.SharePoint.Workflow.SPNoCodeXomlCompiler.LoadXomlAssembly(SPWorkflowAssociation association, SPWeb web)     at Microsoft.SharePoint.Workflow.SPWinOeHostServices.LoadDeclarativeAssembly(SPWorkflowAssociation association)     at Microsoft.SharePoint.Workflow.SPWinOeHostServices.CreateInstance(SPWorkflow workflow)     at Microsoft.SharePoint.Workflow.SPWinOeEngine.RunWorkflow(SPWorkflowHostService host, SPWorkflow workflow, Collection`1 events, TimeSpan timeOut)     at Microsoft.SharePoint.Workflow.SPWorkflowManager.RunWorkflowElev(SPWorkflow workflow, Collection`1 events, SPWorkflowRunOptionsInternal runOptions)

and 

08/23/2012 08:51:01.55 OWSTIMER.EXE (0x0CBC) 0x0DB8 SharePoint Foundation Workflow Infrastructure 98d8 Unexpected Microsoft.SharePoint.SPException: Configuration system failed to initialize     at Microsoft.SharePoint.Workflow.SPNoCodeXomlCompiler.SubCompiler.DoCompile(WorkflowCompilerParameters parameters, String xomlSource, String assemblyName, CompilationPacket& packet, DirectoryInfo& tempDir)     at Microsoft.SharePoint.Workflow.SPNoCodeXomlCompiler.SubCompiler.DoCompile(WorkflowCompilerParameters parameters, String xomlSource, String assemblyName, CompilationPacket& packet, DirectoryInfo& tempDir)     at Microsoft.SharePoint.Workflow.SPNoCodeXomlCompiler.<>c__DisplayClasse.b__c()     at Microsoft.SharePoint.Utilities.SecurityContext.RunAsProcess(CodeToRunElevated secureCode)     at Microsoft.SharePoint.Workflow.SPNoCodeXomlCompiler.DoCompileNewAppDomain(WorkflowCompilerParameters parameters, String xomlSource, String assemblyName, SPWeb web, CompilationPacket& packet, DirectoryInfo& tempDir)     at Microsoft.SharePoint.Workflow.SPNoCodeXomlCompiler.CompileBytes(Byte[] xomlBytes, Byte[] rulesBytes, Boolean doTestCompilation, String assemblyName, SPWeb web, Boolean forceNewAppDomain)     at Microsoft.SharePoint.Workflow.SPNoCodeXomlCompiler.LoadXomlAssembly(SPWorkflowAssociation association, SPWeb web)     at Microsoft.SharePoint.Workflow.SPWinOeHostServices.LoadDeclarativeAssembly(SPWorkflowAssociation association)     at Microsoft.SharePoint.Workflow.SPWinOeHostServices.CreateInstance(SPWorkflow workflow)     at Microsoft.SharePoint.Workflow.SPWinOeEngine.RunWorkflow(SPWorkflowHostService host, SPWorkflow workflow, Collection`1 events, TimeSpan timeOut)     at Microsoft.SharePoint.Workflow.SPWorkflowManager.RunWorkflowElev(SPWorkflow workflow, Collection`1 events, SPWorkflowRunOptionsInternal runOptions) 3010f8ef-7f1c-4adf-abba-7abfd48a28db

Now if the same workflow tried to use the same failing server to process the request the length of pause of our original 5 minute would grow experientially leading to the timer job  in some cases not even attempting to wake to workflow for days.

After reviewing the workflow counter in perfmon for this server it became apparent that no workflow has ever run properly on the this troubled server. This lead to step one of helping resolve this problem which was turning off the "Microsoft SharePoint Foundation Workflow Timer Service" on the troubled server. This stopped new occurrences [Workflow Name]failed to run in the workflow history. Happy me.

Step 2 how ever is how do we get the workflows to run on the troubled server? this still remains a mystery. Maybe ill find the answer for a different post.