Continuous Delivery: Continued

2 years ago, in the article “Continuous Delivery & Sitecore: our implementation”, I presented our system for the automatic delivery of Web applications based on Sitecore. These were the first small steps towards a world of complete automation.

Over the past time, the project has not only not died, but has grown and matured. Moreover, it is still developing and incorporating more and more tools and capabilities. Perhaps the only thing left unchanged is the version control system (GIT) and the branch system.

Major changes that befell the project (I will dwell on each of them in detail):

• SSH delivery is basic only for projects running on Linux and Windows 2003. Projects that host on Windows 2008 R2 and later use Web deploy.
• Since in addition to web applications on CMS Sitecore, we began to develop web applications, services and applications on Umbraco, Episerver and just ASP.NET - now the CD also works with them
• The internal Nuget gallery and updates to our open-source libraries on Nuget.org are also delivered via TeamCity.

SSH delivery of Linux php projects

In fact, everything is as simple as that. Since PHP does not require compilation, the delivery is to pick up the sources from Gita, and picking them up via SSH to the target server.
In TeamCity there is a special plug-in for this - deploy-runner, which, in fact, is responsible for all delivery. In the future, we plan to attach integration and unit tests to this type of builds. Currently used only for internal testing.

Nuget, SymbolSource, packaging and updating

Inside our network, we deployed a Nuget gallery (fork from https://github.com/NuGet/NuGetGallery ), a SymbolSource server for this gallery ( http://www.xavierdecoster.com/setting-up-your-own-symbolsource-server -step-by-step ). Since these are also web applications - Teamcity is responsible not only for their content, but also for their updates (it monitors our fork of repositories, and updates the fork - delivers updates to the web application using the steps described below).
TeamCity is also responsible for publishing Nuget packages to the gallery (both internal and external), updating packages using TeamCity's built-in tools for packaging and publishing packages. Each package is also published in the corresponding SymbolSource gallery (internal or Symbolsource.org).

MsBuild, MsDeploy and everything that revolves around ...

Actually, the delivery itself and everything related to it.
I would like to say a few words about MsDeploy and the reason for leaving SSH. Despite the fact that delivery via SSH is quite fast, preparatory operations require a lot of time and expensive disk operations. Plus, we should always in any build either differentiate the files for delivery (in order to reduce the size of the sent package), or always deliver the entire application (which does not quite positively affect the published package). If we take MsDeploy, which is called by Visual Studio or MsBuild, then MsDeploy is responsible for delivering files to the web application, it also evaluates whether it is necessary to update / delete / leave a specific file, which positively affects the speed of delivery. Also, in the default configuration,
To ensure stable and controlled delivery, the whole process revolves around the MsBuild engine, the behavior of which is controlled by a set of imported projects (targets files), unified by the procedures that are carried out and differing only in the type of project (Sitecore, Umbraco, Episerver or a web application without CMS).
I will dwell on target files in more detail ...

Sitecore

The sitecore.targets file describes the following steps:

1. Preparing the application for the delivery of logical structures to the database - publishing a configuration file that allows the use of any characters and disables the most resource-intensive operations performed by Sitecore during publishing. This step requires a restart of the application and therefore is called only if we have something to deliver to the database
2. Animation of the application using Wget after restart (in fact, just a request to the main page of the site)
3. Adding the latest build assembly that implements CI to the application
4. Static transformation of the web.config file for cleaning garbage that remains in the temp folder after delivery using Sitecore itself
5. Import the sharedtargets file, which is responsible for all other operations during the build, using its own targets. And the sequence of goals in the sharedtargets file is controlled using the AfterTargets and BeforeTatgets directives.
6. Which folders to delete during the build of the application (the list can be expanded in the csproj file of the application itself) is necessary because we do not use self-cleaning of files (SkipExtraFilesOnServer = true), so as not to clear the content files potentially created by the application

EpiServer

The Episerver.targets file describes the following steps and delivery requirements:

1. Adding the latest build assembly that implements CI in the application
2. Import the sharedtargets file, which is responsible for all other operations during the build, using its own targets. And the sequence of goals in the sharedtargets file is controlled using the AfterTargets and BeforeTatgets directives.
3. What folders to delete during build application

Umbraco

The Umbraco.targets file describes the following steps and delivery requirements:

1. Adding the latest build assembly that implements CI in the application
2. What folders to delete during build application
3. Import the sharedtargets file, which is responsible for all other operations during the build, using its own targets. And the sequence of goals in the sharedtargets file is controlled using the AfterTargets and BeforeTatgets directives.
4. Updating the application version in ClientDependency.config (Umbraco cleverly caches static resources)

Build itself

All builds are unified and can only differ according to the requirements of CMS or the consumer. Here I will try to parse all the steps in order (I will indicate in parentheses at the beginning of the description of the step if this step is specific to a target)

1. Informational e-mail sent to the group that the build is starting. It contains the following information:
   - for which project and what environment the build
   is launched - who launched the build
   - what changes are included in the build (comment on checks in the repository, if the comment starts with # number , then # number is framed by a link to the request in our tracking system )
2. (Sitecore) Checks if there are packages in the build (logical elements of the Sitecore database for delivery). If there is, we update the application (put a special configuration file, update the CI assembly) and revive it.
3. (Sitecore) Delivery of packages via WCF service
4. (Sitecore) Publish using WCF service
5. (Microsoft Network Load Balancer) If the application is hosted on several servers and the traffic is balanced using Microsoft NLB, then in this step we turn off the application from the balancing using Drainstop (the new traffic does not get to the application. We finish the service of those users who were at the moment start drainstop)
6. Delivery of App_offline.htm to the root of the application (so that users are not afraid of errors) and disabling cloud monitoring services (so that Uptime does not crash, since the ASP.NET application with App_offline.htm in the root, although it shows a beautiful picture and inscription, returns HTTP status 503). Which monitoring services to disable - controlled by build parameters
7. Nuget restore - although it is possible and necessary to enable Nuget restore at the solution level, some builds change the build process itself (for example, Microsoft.Bcl.Build)
8. MsBuild Publish - this is where all the magic happens ...
   I will not describe everything that MsBuild does, but only describe the changes that are made by sharedtargets.target file.
   1) If the application-specific target file adds CI files - they are added at the very beginning (we virtually mark them as Content with metadata CopyToOutputDirectory = Always)
   2) AddVersionTxt - adds the version.txt file to the root of the web application, which contains build number and date (convenient for testers and third parties - what is now published). In Sitecore it is also visible in the application information
   3) (Sitecore) Static transform for deleting files from temp folder
   4) Encrypt - if an RSA key is added to the configurable folder in the application and the EncryptWebConfigOnTeamcityBuild = true flag is set, then during the build TeamCity will encrypt the appSettings and connectionStrings sections in web.config (you can add sections at the csproj level of the application file)
   5) If the application an additional database is used - then SqlUpdaterSupportTarget is called. As well as the second step, it checks whether there have been changes in the configured folder and, if so, by means of MsDeploy updates the database configured in TeamCity. Able to work with both MSSQL and MySQL. So far, he knows how to update only one database. If there are several, add an additional step before deleting App_offline.
9. (EpiServer) Since Episerver puts the files necessary for CMS outside the root of the web application into the AppData folder and goes to them through VirualPathProvider - we need to check if there were changes in this folder, and if so, upload them to the server
10. Removing App_offline, revitalizing the application and turning monitoring services back on
11. Animation of a web application - Wget runs through all the pages of the first level of sites specified in the build configuration.
    Soon, instead of configuring the build, this task is planned to be submitted to the WCF service.
    Also, at this step, the numerical values ​​are saved (how much time it took). In the future, we will conduct a very rough analysis of the performance of the web application (lamer, but metric :))
12. Analysis of published web.config using OWASP (report is available in Teamcity reports)
13. Standard checks of the web application and sending email with the results of the checks and build to the group.
    At the moment, standard checks include only checking for robots.txt, analyzing it using the parser https://code.google.com/p/robotstxt/ , and also validating sitemap.xml.
    This step should and will be expanded by various integration and unit tests.

In addition to web applications, we are also developing additional services for Windows. And they also need to be delivered ...
There is also a separate build for this, consisting of the following steps:

1. Nuget restore
2. Build with Visual Studio (Visual Studio (sln) runner)
3. Stopping and removing services on the target server using the MsDeploy runCommand provider
4. File upload with MsDeploy
5. Installing services on the target server using MsDeploy runCommand

Well, one more build is necessary for TeamCity to build all the tools described above for themselves. They are all collected in one solution, and by changing it, a build is launched, consisting of the following steps:

1. Nuget restore
2. Visual studio build
3. Powershell script that expands the results of step two to all agents

In addition to the data builds that deliver the code, there is another TeamCity build - not related to the delivery of the code, but just a service one. It consists of two steps and its only purpose is to call the recycle of the process serving the web application. It is needed in order to be able to quickly restart the web application

1. MsBuild, using the recycleApp provider integrated in MsDeploy, causes the application to restart
2. Wget in spider mode loads the first page of a web application (to revive it as soon as possible)

This "build" is not called often, as this is an extreme measure.

Visualization

As a final touch, using the open-source Gource tool , I added a visualization of the CI development: