Visual Studio Online: Continuous Integration and Testing

Original author: Lee Stott
  • Transfer
  • Tutorial

Visual Studio Online , like Team Foundation Server 2015, has the ability to implement a continuous integration process.
In this article, we will look at an example of using Visual Studio Online (VSO) with a Git repository, as well as ways to configure continuous integration, testing, and automatic deployment processes.
Setting up the Visual Studio Online service is one of the fastest ways to organize and plan the process of building and deploying applications for various platforms. The service will deploy and launch within a few minutes on our cloud infrastructure without the need to install or configure on a separate server.

Using VSO and Git

Creating a project in VSO and initializing a remote Git repository

Let's create a new project for the team to work in VSO. We go to the VSO website, go to the main page and click New:

Indicate the name and description of the project. Choose the necessary template for conducting the development process.
Here we select Git as the version control system and click the Create Project button:

When the project is created, click the Navigate to project button: The

main page of the project for the work of our team will be displayed. Now you need to initialize the Git repository. Go to the CODE page and click "Create a ReadMe file". The repository is initialized and the master branch is created. For example, I will configure continuous integration on this branch:

In the screenshot, we see the master branch with the file:

Opening a project in Visual Studio, cloning a Git repository and creating a solution.

Now open the project for the team to work in Visual Studio and clone the repository to create a local copy.

Go to the main page of the project in VSO and click "Open in Visual Studio":

Visual Studio opens a connection to the VSO project.
In the Team Explorer window, enter the address of the local repository and click the Clone button:

Now click “New” to create a new solution:

Select the ASP.NET Web Application project template, specify the project name and click “OK”:

Select the ASP.NET 5 Preview template Web Application and click on “OK”:

Now add a project containing unit tests. Right-click on the solution in Solution Explorer, select "Add New Project" and select the Unit Test Project template. My project is called CITest.Tests.
The solution is as follows:

The UnitTest1 class was created with the only TestMethod1 method. TestMethod1 will be passed because it has no implementation.
Add the second TestMethod2 method, with the expression Assert.Fail. The second method will not be completed and will just show that the test runner has successfully identified and run the necessary tests.
using System;
using Microsoft.VisualStudio.TestTools.UnitTesting;
               Assert.Fail("failing a test");

Save the changes and run the assembly.
Now we need to commit the commit to the local repository and send from local to remote. In order to do this, go to the Changed tab in the Team Explorer window, add a comment to the commit, and select the “Commit and Push” option.

The master branch of the remote Git repository now contains a solution that includes a web application and a project with tests.
Creating a build process definition
Now create a Build Definition in VSO.
We’ll go to the “BUILD” page of the project for the team to work and click the “+” button.

Select the Visual Studio template and click OK.

The Visual Studio template for defining an assembly contains 4 assembly steps:

  1. Visual Studio Build - Solution Build
  2. Visual Studio Test - running tests
  3. Index Sources & Publish Symbols - Indexing Source Code and Publishing Symbols in .pdb Files
  4. Publish Build Artifacts - publish assembly artifacts (dlls, pdb, and xml documentation files)

Now let's leave it as it is, by default, click “Save” and specify a name for the created definition.

To test, click “Queue build”:

Accept the default settings and click “OK”.

We get into the browser of the build process (BUILD-> Explorer). The assembly is queued and, as soon as the process is started, we will see information about it in the output window:

The assembly failed at the Build Solution step with the following error - The Dnx Runtime package needs to be installed.
The reason for this error is that we start the build process on a remote server and we need to install the appropriate DNX environment, which our solution will choose for assembly.
Let's go back to Visual Studio and add a new file to our solution. Call Prebuild.ps1 and add the following Powershell script to it:
    # load up the global.json so we can find the DNX version$globalJson = Get-Content -Path $PSScriptRoot\global.json -Raw -ErrorAction Ignore | ConvertFrom-Json -ErrorAction Ignore
     $dnxVersion = $globalJson.sdk.version
     Write-Warning "Unable to locate global.json to determine using 'latest'"$dnxVersion = "latest"
   # install DNX# only installs the default (x86, clr) runtime of the framework.# If you need additional architectures or runtimes you should add additional calls# ex: & $env:USERPROFILE\.dnx\bin\dnvm install $dnxVersion -r coreclr
    & $env:USERPROFILE\.dnx\bin\dnvm install $dnxVersion -Persistent
    # run DNU restore on all project.json files in the src folder including 2>1 to redirect stderr to stdout for badly behaved tools
   Get-ChildItem -Path $PSScriptRoot\src -Filter project.json -Recurse | ForEach-Object { & dnu restore $_.FullName 2>1 }

The script starts DNVM, determines the version of DNX from the global file . json of our solution, installs DNX and then restores the project dependencies contained in the project.json file.
After adding the Prebuild.ps1 file, the solution will look as follows:

We commit the changes to the local repository and send it to the remote one.
Now you need to add the Powershell script step to the assembly definition.
Let's go back to VSO and edit the build steps. Click "+" and add a Powershell step.

Let's move the task with the Powershell script to the top of the list of build steps, so that it starts first. Click "Script" and select the file Prebuild.ps1. Click “Save” and then “Queue build” to test:

Now all the assembly steps have been completed successfully:

Nevertheless, if we decide to look in more detail at the results of the Test step, we will see a warning - No results found to publish. But we added 2 methods to the solution.
The answer is contained in the following “Executing” expression, which shows that vstest.console was executed for 2 test files - CITest.Tests.dll, which is good. And Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll, which is bad.

You must change the test assembly step Test to exclude the UnitTestFramework.dll file.
Edit the assembly definition, select the Test step and change the Test Assembly path from the current ** \ $ (BuildConfiguration) \ * test * .dll; -: ** \ obj \ ** to next ** \ $ (BuildConfiguration) \ * tests.dll; -: ** \ obj \ **.
Click “Save” and “Queue Build”:

The assembly failed. But this is exactly what we were aiming for. TestMethod2 contains the expression Assert.Fail () and therefore we provoked the failure of the Test step, as can be seen in the screenshot. We caused this failure to prove that the tests run correctly:

Configuring continuous integration, trigger for assembly and deployment of artifacts

Now in the assembly definition, we have a preliminary step that downloads DNX, then a step that provides a solution assembly and a Test step that fails, thanks to the TestMethod2 method.
Now it's time to set up continuous integration and make changes to the UnitTest1 class. Then we commit and submit the changes to the repository, which should serve as a trigger to start the assembly.
Edit the assembly definition and go to the Triggers tab. Select Continuous Integration (CI) and click “Save”:

Edit the UnitTest1.cs file in Visual Studio and delete the TestMethod2 method. Then we commit and submit the changes.
Let's go back to VSO and go to the BUILD page. In the Queued list, we should see the assembly awaiting launch:

All steps completed successfully.
DNX has been successfully installed on the remote host where the assembly is being performed. The decision has come together. All tests started and passed. Character files are generated. Finally, assembly artifacts are published.
That is, we have a new, tested assembly of the application ready.
Deploy build artifacts to a web application server

If we host our web application in Azure, we must add the Azure Web Application Deployment step to the build process definition, in which case the build artifacts will automatically deploy to Azure when the application is successfully built and tested.
Alternatively, we can manually download the assembly artifacts and then copy them to the desired server. In order to do this, you need to go to the list of completed assemblies and open the necessary one. Then select the Artifacts tab and click “Download”. The resulting .zip file will contain the necessary files.

So, we have a website. Let's see how you can test it.
As preparation for running build definitions, we mean preparing the build agent. The link provides steps for configuring the assembly agent.
Create a build process definition and select a Visual Studio

template: Visual Studio template selection automatically adds the Build Task and Unit Tesk Task. Now fill in the parameters necessary for each of the tasks. The Build task takes a decision that needs to be built and configuration options. As mentioned earlier, this solution contains application code, code for unit tests, as well as automatic Selenium tests that we want to run, as one of the stages of assembly validation:

At the end, add the required parameters required for the Unit Test - Test Assembly task and the Test Filter criterion. Note that in this task we take the dll of unit tests, enumerate all the tests and run them automatically. You can enable the Test Filter criterion and use the filter by type of test set if you need to run specific tests. Another important point is that unit tests in the Visual Studio Test Task always run on the server where the assembly takes place and do not require additional settings for deployment:


Using Visual Studio Online for Test Management

Configure virtual machines to deploy the application and run tests

Once the assembly is completed and unit tests have passed, the next step is to deploy the application (website) and run the functional tests.
To do this, the following components are required:

  1. Ready and configured Windows Server 2012 R2 with IIS installed to deploy a website or Microsoft Azure Website.
  2. A set of virtual machines with installed browsers (Chrome, Firefox and IE) for automatically running Selenium tests on these machines.

You must ensure that these virtual machines are configured with remote access through Powershell .

Once the virtual machines are ready, go to the Test Hub-> Machine page to create the necessary configuration as shown in the screenshot. Specify the name for the group of virtual machines and the FQDN / IP address of the IIS / Web server of the virtual machines. It is also necessary to enter the administrator login and password for future settings:

For the test environment, specify a name and add all the IP addresses of machines that have already been configured with the necessary browsers. As mentioned earlier, the automated test system is capable of distributed execution of all tests and can be scaled to a different number of virtual machines.
As a result, in the hub of virtual machines there will be one application in the test environment and the test environment itself, for example, in this example, it is “Application Under Test” and “Test Machines”, respectively.
Application Deployment and Testing Settings

Now we will add a task for deploying the application to a web server and a task for running integration tests on virtual machines remotely.
We will use the same assembly definition by simply adding new steps to implement continuous deployment.
Web site deployment using Powershell

First, copy all the website files to the desired location. Click “Add build step”, add the task “Windows Machine File Copy” and fill in the parameters for copying files. Now add “Run Powershell on Target Machine Tasks” to the definition for deploying / configuring the application environment. To deploy a web application on a web server, select “Application under Test”, as a group of virtual machines, we configured them earlier. We select the Powershell task for the deployment of the website. Make sure that this script is included in the solution / project. This task will execute a Powershell script on a remote machine to configure the website and other additional steps:


Copying tests to the test environment

Add the “Copy Files” task to the assembly definition to copy all test files to the “Test Machines” group.
You can choose any location, the example uses “C: \ Tests”

Setting Up Visual Studio Test Agent

To perform actions on remote machines, first you need to create and configure a test agent. To do this, create a task in which we write information about the remote machines.
Unlike previous versions of Visual Studio, now you do not need to manually copy and configure the controller and test agents on test environment machines:

Running tests on remote virtual machines

When the setup is complete, the final task will be to add “Run Visual Studio Tests using Test Agent” to run the tests. In this task, we specify the information of the Test Assembly and a filter for performing tests. As part of the assembly verification, we want to run P0 Selenium tests, so we need to filter the assemblies using SeleniumTests * .dll as a test assembly.

We can include a runsettings file with tests and any test run parameters, as input. Below we provide data for tests on where the application was deployed using the $ (addurl) variable:

As soon as tasks are added and configured, you must save the build process definition.
Queuing an assembly, running tests, and analyzing test results.

Now that a set of all tasks is ready in the definition of the assembly process, you can start by placing the assembly in a queue. Before that, you need to make sure that the build server and the pool of test machines are configured correctly.

Once the assembly is complete, VSO provides an overview of the assembly results with all the necessary information for subsequent decision making.
The build results contain the following information:

  • The results of passing the steps and highlighting them in color on the left side of the panel, as well as details on the right.
  • By clicking on each step, detailed log entries are available.
  • From the test results, you can see that all unit tests were passed, but there are problems with integration tests.

Clicking on “Test Results” allows you to go to the test run results.
The Test Run page contains a set of graphs and mechanisms for a deeper study of the results:

When you go to "Test Results", you can also see information for each test - the name of the test, the parameters of the test, the author, the virtual machine where it was run, etc.
For each failed test, the “Update Analysis” option is available to analyze it. The following screenshot shows that IE Selenium tests failed. If you click Create Bug, then all test information will be automatically added to the created element:

Settings for continuous integration

Now that all the tests have been examined and defects have been registered, you can configure the definition of the assembly process to automatically start the continuous integration process, unit tests and integration tests for each check-in.
There are two ways to configure:

  • Select “Continuous Integration” to start the process for each check.
  • Select a special scheduled launch to check after all changes are made.

You can choose both options, as shown in the screenshot:

Thus, using a similar definition of the assembly process, we were able to configure the continuous integration process to automatically start the assembly, then run unit tests and validate the assembly.


Ultimately, with the help of Visual Studio Online, we were able to solve several problems:

  1. Create a simple build process definition that includes unit testing and automated tests
  2. Set up a test environment and test agents
  3. Got a report on the results of the assembly and launch of tests
  4. Customized the build process using a continuous integration approach

useful links

Also popular now: