August 4, 2016 at 13:16Fighting Mysterious MSBuild Crashes on XamlTaskFactory
Our team is developing a cross-platform application kernel that should be built on Windows under Visual Studio 2015, Linux with gcc 4.9+, MacOS, iOS, Android, and Windows Phone 8.1+. To automatically check the code on Jenkins, assemblies are configured for all the required configurations. The task of assemblies is to catch code that is not collected on one or several of the platforms or does not pass unit tests and does not allow it to get to the teams of the final applications before making the corresponding corrections. Such a CI process allows the developer to locally use a convenient operating system and development environment, be it Visual Studio, Xcode, QtCreator or even vim + ninja, without being afraid that his changes will not be collected or tests will be thrown in another environment.
In an ideal world, the red assembly on Jenkins (it is he who is used in our role as a build server) indicates a problem in the code. Seeing a red light on the monitor hanging in the corner of the room, the “assembly assistant” should go and fix the problem. In reality, the reasons for the fall of the build can be very different, for example, a break in the connection with the node on which the compilation took place, running out of disk space or the arrival of aliens. Such false positives take away unnecessary time from the team, dull the attention and generally reduce confidence in the CI in the team. I want to tell the story of the struggle against one of these problems.
The problem was specific to MSBuild and showed up with something like this in the log:
For some time, the problem did not appear often, once every few days, and made me curse and once again restart the fallen build. But after moving from virtualoks to new shiny hardware nodes, the situation worsened, random drops could occur several times a day. The situation was finally unacceptable for a long time to build the project (dozens of minutes, with which, by the way, we fought in parallel). Sometimes it was necessary to run an urgent fix through CI, but after waiting a lot of time it was possible to catch a fall, and then you had to wait again.
So, what did lead to the error?
To generate projects, we use gyp , in which there are 2 ways to call an external command during build - these are actions and rules. Actions are implemented through CustomBuild inside vcxproj files.
Example from the documentation:
And with them everything is fine, they do not explode. Rules use a different mechanism. The comment in the code reads:
How it works? For each such rule, MSbuild in% TEMP% generates a C # source (.cs file), from which it tries to compile a dll and use it right there, and if it doesn’t work, it throws an exception .
The comment says:
Indeed, in the system log a couple of seconds before the time of the error (according to the buildserver log), you can find something like this C # compiler error record:
A search for similar errors in Google suggested that it was the size of the Desktop Heap for non-interactive sessions. Indeed, it was like this: the error code matched, and the Jenkins slave agent worked as a Windows service.
Taking this hypothesis into development, I began to play with the value of the SharedSection section in the registry entry HKEY_LOCAL_MACHINE \ System \ CurrentControlSet \ Control \ Session Manager \ SubSystems \ Windows. Along the way, I accidentally managed to make the assembly prone to crashes with almost 100% probability, which somewhat facilitated the iteration of debugging. After reading a little more, I got to the “Enable desktop interaction” checkbox in the Jenkins service properties, and then to the NoInteractiveServices option in HKEY_LOCAL_MACHINE \ SYSTEM \ CurrentControlSet \ Control \ Windows. But all these attempts did not bear fruit. Sometimes builds passed, but they could not catch patterns.
Continuing to tinker with the features of running processes from under the Jenkins service, I came across the following texton stackoverflow. The author talks about the peculiarities of the default behavior of MSBuild when specifying the / M option for it to build several projects in parallel. The bottom line is that MSBuild creates the right number of copies of itself - nodes waiting for tasks. During the assembly process, tasks are scattered on these nodes and are executed in parallel. After the assembly is completed, the nodes are not quenched and continue to wait for new tasks. This also happened on Jenkins, after the build was complete, MSBuild processes continued to hang in memory.
I started to experiment. Having reproduced the fall of the build several times in a row, I killed all the MSBuild processes that were in my memory, and, lo and behold, the next build was successful! Then I armed myself with the instruction with StackOverflow and added to our build script setting the variable MSBUILDDISABLENODEREUSE and forwarding the / nr: false option to the MSBuild call. After that, all MSBuild processes began to die at the end of the assembly, and did not remain in memory.
The solution turned out to be working. Almost 2 weeks have passed, the problem has never been reproduced. And although I did not fully understand the underlying causes of the error, I was able to find a solution that worked and, I hope, will help someone else.
In an ideal world, the red assembly on Jenkins (it is he who is used in our role as a build server) indicates a problem in the code. Seeing a red light on the monitor hanging in the corner of the room, the “assembly assistant” should go and fix the problem. In reality, the reasons for the fall of the build can be very different, for example, a break in the connection with the node on which the compilation took place, running out of disk space or the arrival of aliens. Such false positives take away unnecessary time from the team, dull the attention and generally reduce confidence in the CI in the team. I want to tell the story of the struggle against one of these problems.
The problem was specific to MSBuild and showed up with something like this in the log:
20:03:56 "D:\jenkins\workspace\task\ws\...\SomeTarget.vcxproj" (default target) (429) ->
20:03:56 (_QtMetaObjectCompilerH target) ->
20:03:56 D:\jenkins\workspace\task\ws\...\SomeQtBasedTarget.targets(52,5): error MSB4175: The task factory "XamlTaskFactory" could not be loaded from the assembly "Microsoft.Build.Tasks.Core". Could not find file 'D:\jenkins\workspace\task\ws\TEMP\fv5nnzin.dll'. [D:\jenkins\workspace\jenkins\workspace\task\ws\...\SomeTarget.vcxpro]
For some time, the problem did not appear often, once every few days, and made me curse and once again restart the fallen build. But after moving from virtualoks to new shiny hardware nodes, the situation worsened, random drops could occur several times a day. The situation was finally unacceptable for a long time to build the project (dozens of minutes, with which, by the way, we fought in parallel). Sometimes it was necessary to run an urgent fix through CI, but after waiting a lot of time it was possible to catch a fall, and then you had to wait again.
So, what did lead to the error?
To generate projects, we use gyp , in which there are 2 ways to call an external command during build - these are actions and rules. Actions are implemented through CustomBuild inside vcxproj files.
Example from the documentation:
Copying readme... $(OutDir)%(Identity) And with them everything is fine, they do not explode. Rules use a different mechanism. The comment in the code reads:
MSBuild rules are implemented using three files: an XML file, a .targets file and a .props file. See blogs.msdn.com/b/vcblog/archive/2010/04/21/quick-help-on-vs2010-custom-build-rule.aspx .
How it works? For each such rule, MSbuild in% TEMP% generates a C # source (.cs file), from which it tries to compile a dll and use it right there, and if it doesn’t work, it throws an exception .
The comment says:
This occurs if there is a failure to compile the assembly. We just pass through because we will take care of the failure below.
Indeed, in the system log a couple of seconds before the time of the error (according to the buildserver log), you can find something like this C # compiler error record:
Faulting application name: csc.exe, version: 4.6.1055.0, time stamp: 0x563c1a09
Faulting module name: KERNELBASE.dll, version: 6.3.9600.18233, time stamp: 0x56bb4ebb
Exception code: 0xc0000142
Fault offset: 0x00000000000ecdd0
Faulting process id: 0x1af4
Faulting application start time: 0x01d1d13dbec0f5bd
Faulting application path: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\csc.exe
Faulting module path: KERNELBASE.dll
Report Id: fc6cf36d-3d30-11e6-8260-0cc47ab21249
Faulting package full name:
Faulting package-relative application ID:
A search for similar errors in Google suggested that it was the size of the Desktop Heap for non-interactive sessions. Indeed, it was like this: the error code matched, and the Jenkins slave agent worked as a Windows service.
Taking this hypothesis into development, I began to play with the value of the SharedSection section in the registry entry HKEY_LOCAL_MACHINE \ System \ CurrentControlSet \ Control \ Session Manager \ SubSystems \ Windows. Along the way, I accidentally managed to make the assembly prone to crashes with almost 100% probability, which somewhat facilitated the iteration of debugging. After reading a little more, I got to the “Enable desktop interaction” checkbox in the Jenkins service properties, and then to the NoInteractiveServices option in HKEY_LOCAL_MACHINE \ SYSTEM \ CurrentControlSet \ Control \ Windows. But all these attempts did not bear fruit. Sometimes builds passed, but they could not catch patterns.
Continuing to tinker with the features of running processes from under the Jenkins service, I came across the following texton stackoverflow. The author talks about the peculiarities of the default behavior of MSBuild when specifying the / M option for it to build several projects in parallel. The bottom line is that MSBuild creates the right number of copies of itself - nodes waiting for tasks. During the assembly process, tasks are scattered on these nodes and are executed in parallel. After the assembly is completed, the nodes are not quenched and continue to wait for new tasks. This also happened on Jenkins, after the build was complete, MSBuild processes continued to hang in memory.
I started to experiment. Having reproduced the fall of the build several times in a row, I killed all the MSBuild processes that were in my memory, and, lo and behold, the next build was successful! Then I armed myself with the instruction with StackOverflow and added to our build script setting the variable MSBUILDDISABLENODEREUSE and forwarding the / nr: false option to the MSBuild call. After that, all MSBuild processes began to die at the end of the assembly, and did not remain in memory.
The solution turned out to be working. Almost 2 weeks have passed, the problem has never been reproduced. And although I did not fully understand the underlying causes of the error, I was able to find a solution that worked and, I hope, will help someone else.
