Infrastructure for load testing in the cloud - with Visual Studio 2013 and only when necessary

This article was created with the invaluable help of our colleague from Kaspersky Lab, Igor Shcheglovitov , QA engineer at the Cloud Research Department. Foreword - by ahriman.

The biggest challenge in stress testing is to start doing it. What do you need to start doing stress testing? In the general case, one machine and a target service. This method is good exactly until the moment when the resources of one machine are no longer enough to generate the required load.

One car has ceased to be enough - we take two. Three. Ten. Where to get ten cars? Ask the IT department. After stress testing, notify the IT department. Two hours after the IT department returns the infrastructure to its possession, [something] is discovered and the infrastructure is needed again. We ask again - and the time that the IT department spent earlier on deploying the infrastructure suddenly increased (of course, no one knows why).

The usual story. Developers want to quickly get and start using. Load testing is an iterative process, and building infrastructure is also. What if developers had this infrastructure all the time? Customized, on request? Off when not necessary to pay the bare minimum? How to deploy this infrastructure based on Visual Studio 2013 - in the article. As a result, you will always have a ready-made infrastructure for load testing, which is enough to include - and you can start testing. In the off state, only the storage for the images of these machines will be paid.




Purpose : To create an infrastructure for load testing. 

Description :
All infrastructure will be deployed in Microsoft Azure virtual machines.
Load testing in Microsoft Azure can be performed in three ways:

1. Visual Studio Online (http://www.visualstudio.com/en-us/get-started/load-test-your-app-vs).
2. Microsoft Azure Cloud Services
3. Virtual machines

In difficult cases, you may need all of the infrastructure participating machines to be combined into a domain or located in a hybrid model with a VPN - we will omit this situation, since the article will be somewhat larger in this case.

Test object
WCF service that provides one synchronous method (calculator) via SOAP. The solution can be used to test projects of any complexity. The service can be hosted both as a Cloud Service and as a regular service in a separate virtual machine. If the service will be hosted on a virtual machine, then during the load we will be able to collect from the machine any desired (configured) performance counters. However, there will be no automatic scaling capability that Cloud Service has. Our service will provide one endpoint for connection, according to basicHttpBinding.

Sequence of actions:
During the laboratory work, 4 virtual machines will be deployed on Microsoft Azure:

• Testing Controller and SQL 2012 Express Server

•  Testing agent

• Test service (SUT - System under the testing)

• Visual Studio 2013, which will be developed load test

 
Controller
Create a controller - when selecting an image on Microsoft Azure, select the Windows 2012 Server R2 image. In the virtual machine settings, when creating, open the following ports:
a) TCP port 445 for remote assembly of performance counters
b) UDP port 1434 for SQL Browser and TCP 1433 for connecting to SQL server
c) TCP port for connecting to Test Controller`y 6901
d) Remote Destop, it is needed on all created virtual machines.



Connect to the virtual machine through RDP (by clicking the Connect button on the Microsoft Azure management portal) and download Agents for Microsoft Visual Studio 2013 . Install TestController (vstf_testcontroller.exe)

After installation, run Test Controller Configuration Tools and specify the account with which you are connecting to the virtual machine. Check the box Configure test controller for load testing .
 

 
Download and install the SQL Server Express distribution from the UI link. Launch SQL Server Configuration Manager and enable Pipe and TCP / IP protocols. In the TCP / IP settings, enable all available Enabled IP addresses and for IPAll set the static port 1433.


 
On the SQL Server Services tab, enable and start the SQL Server Browser and SQL Server services.

In the Firewall settings, allow connections to the following ports:
a) outgoing connections to the agent (port 6910)
b) incoming connections to the 6901 controller service
c) incoming connections to the RPC service for collecting performance counters - port 445
d) connections to the studio (LoadTest framework), outgoing port 6915
e) incoming connections to TCP port 1433 and UDP 1434

For simplicity, you can write and execute a bat file:
 
netsh advfirewall firewall add rule name = «Test Agent out » dir = out action = allow protocol = TCP localport = 6910
netsh advfirewall firewall add rule name = « Test Controller in» dir = in action = allow protocol = TCP localport = 6901
netsh advfirewall firewall add rule name = "RPC In" dir = in action = allow protocol = TCP localport = 445
netsh advfirewall firewall add rule name = "VS Studio Out" dir = out action = allow protocol = TCP localport = 6915
netsh advfirewall firewall add rule name = “SQL Express” dir = in action = allow protocol = TCP localport = 1433
netsh advfirewall firewall add rule name = “SQL Express” dir = in action = allow protocol = UDP localport = 1434

Open the registry branch HKEY_LOCAL_MACHINE \ SYSTEM \ CurrentControlSet \ Control \ Lsa and add the DisableLoopBackCheck parameter (value 1) to the DWODR there. This solves the problem of connecting to the SQL server from an external domain. For the testing agent to be able to connect to the controller, you need to set the DNS suffix in the IPv4 settings (network adapter):
 

 
Go to Test Controller Configuration Tools. In the line for connecting to the SQL database, write instead of localhost the full DNS name of the virtual machine and click Apply settings.
 

 
The test controller configuration process begins. The last message will be warning, which is not worth paying attention to. 
Agent
Create a new virtual machine. When creating, open the following ports:
a) TCP port 445 for remote assembly of performance counters
c) TCP port for connecting to Test Agent 6910
d) TCP port for Remote Desktop

Connect to the created machine via RDP (by clicking the Connect button) and download the distribution kit "> Test Agent . Install the Test agent.
 
In the Firewall settings, you should allow connections to the following ports:
a) Incoming connections to the 6910 controller service.
B) Outgoing connections to the 6901 testing controller
c) Incoming connections to the RPC service to collect performance counters - port 445
d) Outgoing TCP connections to the port that the test service listens to, in our case it is 9117.

bat - file: 
netsh advfirewall firewall add rule name = “Test Agent In” dir = in action = allow protocol = TCP localport = 6910
netsh advfirewall firewall add rule name = "Test Controller Out" dir = out action = allow protocol = TCP localport = 6901
netsh advfirewall firewall add rule name = "RPC In" dir = in action = allow protocol = TCP localport = 445
netsh advfirewall firewall add rule name = "Test service OUT" dir = out action = allow protocol = TCP localport = 9117 

Enter the suffix cloudapp.net in the DNS settings of the network adapter.
Run the Configuration Test Agent, (a suggestion appears that the agent can be launched as a service or desktop application, select the service). Enter your account.

Enter the connection string to the test controller (DNS + port) and click Apply.
If during the configuration of the Test Agent it gives an error that it is not possible to establish a connection between the controller and the agent, but the log shows that the agent was able to connect to the controller, then in the controller configuration (on the appropriate machine), in the appsettings section, specify the setting:
/> 
( details ) 
At this step, communication should be established between the agent and the controller. 

Test Machine Virtual Machine

The WCF service was selected as the test object.
Interface: 
    [ServiceContract]
    public interface ICalculator
    {
        [OperationContract]
        int Sum (int a, int b, int timeOutInMiliseconds);
    }
 
The source code along with executable binary files can be taken here (CalculatorService project). Create a new virtual machine. When creating, open ports:
9117. This port listens on our test service.
RPC port 445. 

After creating, open the same ports in the Firewall operating system.

Next, using the sc create NewService binpath = ".... \ Debug \ CalculatorService.exe commandcreate and run a test service from the services snap-in (services.msc).
 
Studio

Create a new virtual machine (Windows 8, with Visual Studio 2013 Ultimate). In the console, open port 6915. This port is needed for the test controller to interact with the studio.

Connect to the virtual machine via RDP.
In the Firewall settings:
a) Open port 6915 for incoming connections
b) Open port 6901 for outgoing to the controller
c) Outgoing ports UDP 1434 and TCP 1433 for connecting to the SQL database
d) Outgoing 445 for connecting to the RPC (collection of performance counters)

Also, as elsewhere, register the cloudapp.net DNS suffix in the IPv4 settings. 
After that, you should gain access to remotely collect performance counters.

To do this, run the command:
net use \\ machineName \ IPC $ { password } / user : { username } / persistent : yes
 
As machineName, you can list all virtual machines — the controller, agent, and the machine with the test service installed. 

Launch Visual Studio 2013. Open Solution from the solution package. Create a new Web performance And Load Test Project solution. Add UnitTestProject1 to it (it contains a simple test method along with a wrapper to call the Calculator test service)
In SolutionItems, add a new file of type TestSettings and open it. On the Roles tab, install RemoteExecution and manually write the DNS name of the controller virtual machine.
 

 
In the Load Test menu, select the Manage Test Controller tab and make sure that the studio can connect to the controller:
 


The connection line to the database and the name of the agent (if it is active) should be visible.
From the Load Test menu, select Select active test settings => Remote.testsettings. Create a new LoadTest. In the Load Test Wizard, on the Test Mix tab, add the TestMethod1 test unit from the UnitTestProject project. If it does not appear, then you need to close the Wizard and recompile the solution.
On the Counter Set tab, add the machine on which the test service is installed (if we want to collect counters from it during the testing process), and select at least one of the default counter sets for it.
 


Save the test. Add WCF performance counters from a test service machine.
To do this, double-click on the created load test. Using the right mouse button create a new Counter Set. Right-click on the created Counter Set and select Add counters. Select the desired computer (from which you need to read the available counters) and the desired counter.
 

 
Click OK, after which you can add the created Counter Set to Counter Set Mapping, so that the system collects counters during the test.
 
Now the test can be run. The test results of each test run will be saved to the LoadTest database automatically created on the controller.

Using MS Office plugins, you can build a very detailed report on load testing (more on http://msdn.microsoft.com/en-us/library/ee923686.aspx ) - compare two starts of any counters relative (response time, errors, memory processor time, etc.) or build the dynamics of each counter according to the range of test runs.
Testing agents can be scaled horizontally. Those. Using the Microsoft Azure Management Console, you can save a virtual machine image with a testing agent and create a clone of it. After starting, it will automatically connect to the controller.
Thanks to this approach, the load can be generated from several agents at once. This is necessary when one agent fails.