Automated tests in the cloud

  • Tutorial
If you test at least a little, then you know how to not just create an environment with a new build for verification. If the solution is not complicated, then you can do everything by hand - erase the old files, clean the registry, database, copy the new build. But if you have a serious multi-tier solution, it is best that all operations are automated. Thus, the influence of the human factor can be reduced. If you find a problem, then just fix the scripts to fix the errors and start again. Team Foundation Server includes a complete set of tools for organizing Lab Management - a test environment management environment. The interesting thing is that you can organize a testing laboratory without any infrastructure investments. All that is needed for this is already in Azure and Team Foundation Service.


Our developers will store the code and manage tasks in the cloud TFS , and they will have the opportunity to create builds in the cloud, as well as deploy these builds on virtual machines for automated testing.

Automated tests mean a test that launches the program itself, clicks on the buttons, and in general repeats the interactive actions of a real tester.


Register at and create a project. After that, we will design our test bench. It will consist of two virtual machines.

  1. Desktop client : the machine on which our desktop application will run and interactive tests will be performed.
  2. Server : the machine on which the services for our desktop client are running, maybe this is a database, maybe a web application.

In addition, we need a build server: a virtual machine on which the project will be compiled from source, and a unit test run.

In order for these machines to interact with each other, the first step in the Azure management portal is to create a virtual network:

Then we create two virtual machines for the stand and one machine for the build server:

When creating the machines, you must specify the virtual network that was created a step earlier.

For convenience, you can take a ready-made virtual machine for the build server; the library has a machine with Visual Studio 2013 Ultimate preinstalled. The rest of the machines are empty, this is Windows Server 2012 or 2008 R2.
In order for our entire cloud infrastructure to function correctly, we need to add additional endpoints for connections between clients (Visual Studio, Microsoft Test Manager), the TFS server, and these machines themselves.

Accordingly, for the rest of the machines should indicate:

  1. ct -lab : 137,138,139,6901,6910
  2. ag -lab : 137,138,139,6910

Additionally, you must ensure that names between these virtual computers are visible on the virtual network. To do this, create (or use an existing) DNS server and register it in our virtual network. For the needs of my script, editing hosts is enough. You can read more about setting up DNS on the Azure portal . If there is no DNS server, then during the configuration of Test Controller and Test Agent, you may encounter the error CreateControllerObject: attempt 0, System.Net.Sockets.SocketException (0x80004005): No such host is known. To resolve it, we prescribe the suffix с in the adapters of DNS machines .


Install ct-lab and ag-lab with Test Controller and Test Agent . In general, it is not recommended to install Test Agent on the Test Controller machine, but doing so is not forbidden. The whole scenario that is presented here for the infrastructure of three computers can be installed on one virtual machine (this simplifies dancing with DNS and endpoints ports).

The ct-lab computer will first of all be Test Controller. After a short installation of the files from the distribution kit, the configuration step will begin:

Accordingly, next we install and configure the Test Agent on the ag-lab machine:

If you did everything correctly, then preliminary checks before configuration will show green marks at all points.

Test Manager Configuration

After you have created the environment infrastructure, you must configure the Lab Environment in Microsoft Test Manager.

Add our virtual computers there (with full names -, specify the roles from the drop-down menu (Desktop Client, Server) and configure the controller:

After the verification of the configuration, after some time (3-5 minutes) you should Ready status will turn on in your environment. The ag-lab machine is configured interactively and the test agent should start there:

Build server configuration.

The finished Visual Studio 2013 Ultimate virtual machine from the Azure library does not contain Team Foundation Server 2013, so we download the web installer from the MSDN website and install TFS on the bc-lab machine. Next, skip the server configuration steps and select only the Build Server configuration item.
We specify the necessary parameters in it and run the build controller and build agents on the bc-lab machine. Unfortunately, Lab Management infrastructure works only with dedicated build controllers, a virtual controller that is part of the Team Foundation Service is not suitable for this purpose (you cannot create more than one build agent).

Build Configuration.

In order for our script to work, it is necessary to create two build definitions tied to the bc-lab controller.

  1. The build build itself : this build takes the latest version of our project, compiles, and runs unit tests.
  2. Deployment and launch test build: Next, the build picks up the baton, copies binary files to test machines, and starts automatic tests.

I think you will not have any problems with the configuration of the first option, we will dwell on the second one in more detail. After filling in the standard parameters (where to get the source from, who is the build controller), you should change the build template on the Process tab:

Click on the Lab Process Settings configuration button and then go through the steps to see the wizard. We specify CloudEnv as the environment name (we created it at the step of working with Microsoft Test Manager) and our compilation build. Next, a crucial step where to specify deployment scripts:

The first script will be run on the Desktop Client role (ag-lab computer), the second on the Server role (this is ct-lab). Scripts should be put in the c: \ tools directory on these machines. The first script takes the $ (BuildLocation) parameter and copies the collected build binaries from the network drive:

SET “inputpath =% 1”
xcopy% inputpath% c: \ Calculator / E / R / Y The

second does nothing, just an Echo parameter, and is given here as an illustration. In fact, we are not deploying anything to Server.
The user on whose behalf the deployment script is run by default is the system account. On the one hand, this allows you to do anything inside the script without fear for access violation, but on the other hand it somewhat limits network interaction.
In order not to have to manually attach the network drive where the build is located, registering passwords in cmd, the easiest way is to redo the parameters of the Visual Studio Lab Agent Service on the machines where the build agent is installed:

The last step of the wizard is the configuration of which tests to run:

The test plan, test configuration and test settings must be created in the Microsoft Test manager.

Create an automatic test

The final step is to create and configure tests. Add the new Coded UI Test to your test project. You can create it from scratch or take as a basis what the testers recorded as Action Recording. After the test is created, it needs to be attached to the test case, this is done in Visual Studio:

After that this test case is added to the test suite (in our case it is called Auto-BVT):

To prevent CodedUITest from starting during the modular testing you can use the categories of tests. Just add the attribute directly in the test code:

And then we filter in the build parameters:

How does it all work together.

Now it’s enough for the developer to initiate the build of the Calculator-BVT build and the following will happen:

  1. Build Calculator-BVT will call the Build Calculator
  2. Build Calculator on the bc-lab controller will receive the latest sources of our project from cloud TFS
  3. Next, the project will be compiled
  4. Unit tests performed
  5. If everything is ok, then control returns to the Calculator-BVT build
  6. This build runs deployment scripts
  7. And then the automatic tests from the test plan

If everything went fine, then in the build report all these steps will be indicated:

What's next.

There are many options for how projects are assembled and tested. For example, your build might build a project for the Azure web role and deploy it automatically. And then you can run automated tests for web projects. In a test environment, there may be more or less machines. However, some of them may even be located on your organization’s network. You can also leave the build server “with you” together with the controller. In general, there are a lot of configuration options, which allows you to get the necessary level of flexibility.

The most important thing is the ability to reduce the cost of configuring the test environment. The more steps are automated, the more tests you can carry out, the less chance of human errors and therefore your decisions will be better.

Also popular now: