Visual Studio Online Testing Tools, Experience, and Manual Comparison

This article continues the cycle started by our friends from Kaspersky Lab and describes the real experience of using Microsoft testing tools with recommendations and conclusions. The author is a testing engineer, Igor Shcheglovitov (@ ins4n3 on the hub). Our testing articles are tagged with #mstesting .

Introduction

In a previous post, we looked at creating an infrastructure for cloud load testing on Microsoft Azure virtual machines. Now we’ll look at a stress testing tool — Visual Studio Online — and compare it with a manual approach.

Load testing using Visual Studio Online allows you to automatically create and configure all the necessary infrastructure in the cloud - deploying a controller and the required number of agents with specific settings. The results of running a particular test always remain in the VS Online cloud base, and you can access them at any time.


Previously, I did not use this approach for 2 main reasons:
- the infrastructure worked unstably and often crashed
- it was not possible to remove performance counters from the loaded system,
i.e. test results were limited to bandwidth data.

In Update 3 at Visual Studio 2013 , it became possible to dynamically load the necessary performance counters from Application Insights telemetry directly during load testing (more details can be found here , however, there have been many changes since general writing only).

We will consider setting Application Insights for a simple WCF service Calculator (from the previous article), and also test it for load using Visual Studio Online.

[ServiceContract]
public interface ICalculator
{
[OperationContract]
int Sum(int a, int b, int timeOutInMiliseconds);
}

We will also compare the new approach with that described in the last post.
We will need:

- Visual Studio Ultimate 2013 (Update 4),
- Visual Studio Online account
- One virtual machine in Microsoft Azure.
- Test data

Download the project used in the article .
(now the host for the WCF service will not be Windows Service, but IIS).

Environment configuration

We create a new virtual machine through the Azure console (I experimented on a Windows Server 2012 R2 image). When creating a virtual machine, open port 80 (HTTP).

After successfully creating a virtual machine, we connect to it through RDP. In order to run the HTTP service on our WCF virtual machine, you should put additional components (IIS, etc.) there. This is done very simply:

• Open Service Manager and select Add Roles and Feature from the Manage menu .
• In the Installation Type step, select Role-based or feature based installation .
• In the Features step, expand the list of features of the .NET Framework 4.5 Features => WCF Services and check the HTTP Activation checkbox . The installer itself will connect the other dependent components of IIS, ASP.NET, etc.
• Start the installation process

After successfully setting up your infrastructure, you should deploy our WCF service.
For this:

• From the folder with test data, copy the Service folder to an arbitrary directory on the virtual machine.
• Open the IIS Management Console ( Server Manager => Tools => Internet Information Service (IIS))
• Create a new Application with a calculator service inside the default website

Further, in order to make sure that everything works correctly, you can open the service directly in the browser - http: //localhost/applicationName/Calculator.svc A

page containing a link to generate wsdl should open

Configure Application Insights

After successfully setting up the virtual machine through the Azure portal, you need to create a new Visual Studio Online account (or use an already created one).

Then go to the VS Online control menu in the browser and click on the Application Insights link.

By default, if no applications have been added to Application Insights, the setup page for a simple test opens, checking the availability of a certain URL address. You can skip this and click the link I want to monitor something else .
After that, for further configuration, you will need to answer a few simple questions (see screen)



Next, click Click here to show instructions.

On the next page, you will need to specify an arbitrary application name and click Create (to be more precise, the name of the virtual container Application Insights, where the diagnostic information will be stored).



After that, the portal will automatically generate ApplicationInsights.config (config file with personal diagnostic settings) and the Microsoft Monitoring Agent installer .
Next, copy ApplicationInsights.config to the root folder of the Calculator service on the virtual machine and install the agent.

Please note that during installation you need to check the box.



After installation, you must restart IIS.

By default, ApplicationInsights.config is configured in such a way that statistics will be collected from only a few basic performance counters. But we can add additional counters to this file - for this it is enough to list them in a special form inside the PerformanceCounters XML section, for example:


In order to find out the correct name of the necessary counters, I recommend downloading them through the perfmon utility . To do this, add the necessary counters to Performance Monitor, select them and save them with the right mouse button in html-format.



If you open the saved file in notepad, then there you can see the correct name of the counters in this form:

Running tests

Open test solyushin in Visual Studio (from test data).
In the menu Studio Team explorer to connect to the generated Visual Studio Online account.



Next, in the solution, open the Remote.testsetting file and select Run tests using Visual Studio Online on the first tab .


Since we have a very simple test, you can skip the additional steps in the testsettings file. Using these steps, you can configure virtual test agents (for example, install certificates, which is important for HTTPS services, run any batch files, etc.).

Next, in the app config of the RemoteTestProject project, you need to specify the correct URI that the WCF Service Calculator listens for.

Now you can proceed to setting up the load test.

By default, the load test will run in 250 threads on 2 single-core agents, i.e. each agent will generate a load of 125 threads.
To increase the number of agents, use the Run Settings => Agent Count ( Total Cores ) setting . You can see some additional settings, as well as possible problems, by the link .

To connect performance counters from Application Insightsright-click on Run Settings => Get Performance Data from Application Insights and select the counters of interest.



When running load tests, Visual Studio Online will start allocating resources for agents.



Once resources are allocated, the load test will begin. Performance



counters from Application Insights during the load test will be on the Application tab.


After the test is completed, the report can be downloaded and viewed in the usual form

Method Comparison

For the experiment, I conducted a load testing of this WCF service using both approaches.

In the first case, I deployed 2 single-core virtual machines and installed Visual Studio Agents on them. Put the local controller and run the test. As a result, two agents were able to generate a load of about 500 calls per second, while the CPU load on both agents reached 100%, while the processor load on a virtual machine with a test service was approximately 15-20% (the main contribution was naturally made by the process w3wp).

I repeated this test, but using Visual Studio Online + Application Insights (Agent total count = 2). The result was identical to the first, 2a single-core cloud agents generated a continuous load of approximately 500 requests per second. After waiting for the test to complete, I downloaded the report and found that the CPUs were also 100% loaded on virtual agents.



An increase in the number of agent cores to 4x showed an almost linear increase in load to almost 800 requests per second (server load averaged 40 %).
Thus, I assumed that our simple WCF service Calculator is able to withstand a load of 3000 concurrent requests per second (working on a single-core machine), taking into account the fact that we still transmit a timeout of 1 ms in the request.
In fact, everything turned out to be wrong. When I started the load test on 10 agents (250 threads), the maximum performance was about 1000 requests per second, while the agents were underloaded (CPU load 30%) and the server was overloaded (CPU 100%).

I began to understand and rewrote my load test, so as to see the growing dependence of the server CPU on the increasing load. Every 10 seconds, the load increased by 5 users.

At around 100 users, the maximum performance is ~ 1000 requests per second, a further increase in load leads to a rapid degradation of the CPU of the Calculator server.

Output

VS Online allows you to raise the cloud infrastructure of load testing, the required power, almost by switching a single button. And it is worth noting - this infrastructure is in no way inferior to the classical one (with agents and a controller). Now about the price. Application Insights is currently in Preview and is free. One minute of a virtual user costs 1 kopeck, i.e. a load test of 1000 virtual users for 10 minutes will cost you about 100 rubles (regardless of which agent you generate a load or 10).