Skip to content

How to become ‘Performance Testing As A Service (PTaaS)’ Ready

2013 June 25
by jonathon.wright

Following on from my previous blog post on “How to become ‘Automation Ready’ I thought it would be nice to apply the same principles but this time with ‘Performance Testing As A Service’ (PTaaS).

 “Cloud-based ‘Infrastructure As A Service’ (IaaS) Feature-Driven ‘Acceptance Test Driven Development’ (ATDD) through’Performance by Example’ using ‘Organisational-wide Agile’ (OA) approach in “Continuous Integration, Build & Delivery” (CIBD)  Environment’ via ‘High Volume Mobile Automated Testing’ (HVMAT)’”

In layman’s executing performance testing in the cloud directly from your mobile spinning up thousands of real end devices (both physical mobile devices & virtual clients) to test against real applications or website’.


The quote from one of my favourite authors  inspired me to write this blog as I’m currently doing this for a global retail client in the states which required performance testing featuring thousands of real end mobile client devices (combination of iPads / Andriod tablets) connecting to the internet (via 3G/4G  or local store WiFi).

“The best time for planning a book is while you’re doing the dishes” – Agatha Christie

Now with only a single sprint complete non-functional testing it was time to practice what I preach:


Bit of background around non-functional testing and the how to define the perfect ‘Non-Functional Requirement‘  (NFR) which I defer to a good friend & colleague Stevan Zivanovic of BTTB who provides a excellent “Non-Functional Requirement Cube” (NFRc):


No stranger to ‘Performance Testing’ the following list of acronyms used within this blog:

Acronym Definition
PTaaS Performance Testing As A Service
PTM Performance Test Management
PTE Performance Test Environment
SLM Solution Lifecycle Management
ALM Application Lifecycle Management
SUT System Under Test
EUT Environment Under Test
NFR Non-Functional Requirement
NFRc Non-Functional Requirement Cube
PTS Performance Test Session
POS Performance Optimization Sessions
PTP Performance Test Profile
PUP Performance User Profiles
PIP Performance Interface Profiles
PBP Performance Background Profiles
BPM Business Process Modelling
BPS Performance Test Scenario = Business Process Scenario
BPT Business Process Test
TA.db Test Asset db
TA.c Test Asset cube
TA.j Test Asset journals
DLT Distributed Load Testing
PPT Protocol-level Performance Testing
HVAT High Volume Automated Testing
HVMAT High Volume Mobile Automated Testing
TFS Team Foundation Server / Service
IaaS Infrastructure As A Service
NAS Network Access Storage
CIBD Continuous Integration, Build & Delivery
DDA Dynamic Data Adapters
AI Actionable Insight
OA Organisational-wide Agile
ATDD Acceptance Test Driven Development

The source for some of this content in this blog was presented a few years back in 2011 at the ‘British Computer Society’ (BCS) ‘Special Interest Group in Software Testing’ (SIGiST) conference (Conference Slides):

So the question is again how to become ‘Performance Ready’ in 4 easy(ish) steps?

Step 1 – Select the ‘Performance Test Scenarios’ (PTS) that will make up your ‘Performance Test Profile’ (PTP) this can be done in a number of different ways:


Option A – Convert existing ‘Business Process Scenarios’  (BPS) which represent paths through the ‘Business Process Model’ (BPM) into ‘Performance Test Scenarios’ (PTS) by re-executing current automation solution and encapsulating ‘Test Asset journals’ (TA.j) or directly from either the ‘Test Asset cube’ (TA.c) or a ‘Test Asset db‘ (TA.db). This source can be exported into a generic ‘WebTest’ compatible format (feature supported by


Option B – Manually generate ‘Performance Test Scenarios’ (PTS) using a tool such as ‘Fiddler’ that can capture all the ‘Http traffic’:


TIP: Once completed ‘Performance Test Scenario’ (PTS) has been captured in ‘Fiddler’ this session can be exported to ‘WebTest’ compatible format:


Option C – Manually generate ‘Performance Test Scenarios’ (PTS) using a Load Testing tool such as ‘Microsoft Load Testing’ (VS2010.2/VS2012.3) or ‘LoadRunner’ (v11.5):


Step 2 – Select the ‘Performance Test Profile’ (PTP) or test transaction mix of ‘Performance Test Scenarios’ (PTS) previously generated:


TIP: The ‘Performance Test Profile’ (PTP) test transaction pacing model can be defined either by taking real figures from monitoring a snapshot of the live ‘Solution Under Test’ (SUT) or defining the volumetrics calculations manually:


Step 3 – Select the ‘Performance Test Session’ (PTS) by selecting one or more ‘Performance Test Profiles’ (PTP) such ‘Performance User Profiles’ (PUP), ”Performance Interface Profiles’ (PIP) such as ‘Batch’ or ‘WebServices’ and the ‘Performance Background Profiles’ (PBP)  such as ”Ambient’ Traffic’ :


The ‘Performance Test Session’ (PTS) also applies a session pattern such as ‘Load/Stress/Soak’ combined with any goals such as ‘Benchmark/Month-End/Peak’ :

TIP: This approach provides full traceability of the ‘Non-Functional Requirements’ (NFR) otherwise referred to as ‘Performance Test Reference’ (PTR):


This approach supports ‘Acceptance Test Driven Development’ (ATDD) through the abstraction to ‘Business Level Question’ (User Stories) that are asked by the business right down to the individual ‘Business Process Tests’ (BPT) or individual ‘Business Test Transactions’ (BTT)  that provide the ‘Business Level Answers’ (Acceptance Criteria) all in a language that the business can understand and make ‘Actionable Insight’ (AI) to make informed business decisions:


Step 4 – Configure the ‘Performance Test Environment’ (PTE) ready to execute against the ‘Solution Under Test’ (SUT) or ‘Environment Under Test’ (EUT) as part of the ‘Performance Test Management’ (PTM) solution.

For this example ‘Performance Test Environment’ will be in the Azure cloud:


Executed against a Multi-Tier ‘Solution Under Test’ (SUT) again hosted in the Azure cloud:


Performance Test Management’ (PTM) solution will be ‘Microsoft Test Manager’  as part of the ”Team Foundation Service’ in the Azure Cloud:


The ‘Performance Test Management’ (PTM) solution allowed access to ‘Microsoft Test Manager’  via both Web and Mobile:

PTaaS_25-06-2013 10-08-23

The ‘Performance Test Management’ (PTM) solution also supported Legacy access via Desktop Client of ‘Microsoft Test Manager’ which requires ‘Visual Studio 2012.3 Premium or above”:

PTaaS_25-06-2013 10-08-28

The ‘Performance Test Environment’ (PTE) requires access to ‘Test Controllers’ and ‘Test Agents’ to execute / run against a Multi-Tier ‘Solution Under Test’ (SUT):


The ‘Test Controller’ needs to be configured for ‘Load Testing’ and de-registered from any ‘Team Project Collection’:


To support ‘High Volume Automation Testing’ (HMPT) for distributed testing we need to support the required virtual users :

Infrastructure as a Service – Test Agent – Instance Resources:

Configuration Component CPU HD Memory
< 1000 virtual users Test agent 2.6 GHz 10 GB 2 GB
< 2000 virtual users Test agent Dual processor 2.6 GHz 10 GB 2 GB
N x 2000 virtual users Test agent Scale out to N agents each with Dual 2.6 Ghz 10GB 2GB

The ‘Test Controller’ also requires additional resources to control the required amount of virtual users:

Infrastructure as a Service – Test Controller – Instance Resources:

Component Test Controller Application Tier Test Controller Data Tier Test Controller Application/Data Tier


Min: 1 GHzRec: 2 GHz Min: 1 GHzRec: 2 GHz Min: 1 GHzRec: 2 GHz

Disk – System

Min: 1 GBRec: 1 GB Min: 1 GBRec: 1 GB

Min: 1 GB

Rec: 1 GB

Disk – Install

Min: 1 GBRec: 48 GB Min: 8 GBRec: 48 GB

Min: 8 GB

Rec: 48 GB

Memory Min: 1 GBRec: 1 GB Min: 1 GBRec: 1 GB

Min: 1 GB

Rec: 1 GB

Depending on the level/detail of monitoring for the  ‘Environment Under Test’ (EUT) is required. In this case the collection ‘Dynamic Data Adapters’ (DDA) such as traditional performance counters (‘perfmon’) of the a Multi-Tier ‘Solution Under Test’ (SUT) that may have a number of Application / Database Servers:

Component Test agent Test controller application tier Test controller data tier Test controller AT/DT
CPU Depending on the test, the CPU is frequently the limiting factor. Not heavily used. Not heavily used. Not heavily used.
Disk Heavily used when detailed logging is enabled in your load tests. Not heavily used. 10 GB space required for 24 hours of test data. 10 GB space required for 24 hours of test data.
Memory Depending on the test, memory might be the limiting factor. Not heavily used. Heavily used by SQL. Heavily used by SQL.

Depending on amount of persistent storage in the cloud you may want to detach the ‘Load Test’ DB and locate it onto cloud based ‘Network Access Storage’ (NAS):


To provide support for “Continuous Integration, Build & Delivery” (CIBD)  you need to create a Test Setting for a ‘Distributed Load Testing’ (DLT) that contains the roles (Dynamic Data Adapters) and deployment (Build Process Template):


Complex ‘Performance Test Session’ (PTS) containing a number of different ‘Performance Test Profiles’ (PTP) such ‘User’, ”Traffic’ or ‘Ambient’ can be executed by a single command (for example):

Microsoft Load Testing ‘Protocol-level Performance Testing’ (PPT):

“Mstest /TestContainer:PerformanceTestProfile_Soak.loadtest /testsettings:Remote.Testsettings /resultsfile:D:\results\MyResults.trx”

or GUI-based Cross Browser (i.e. IE, Safari, Chrome, FireFox) ‘High Volume Automated Testing’ (HVAT) :

“ArtOfTest.Runner.exe list=”MasterDriver.aiilist”

or GUI-based Cross Platform (i.e. WP8, Andriod, iOS, BlackBerry)  ‘High Volume Mobile Automated Testing’ (HVMAT):

mstest /testlist:MasterDriver /testmetadata:”TestProject.vsmdi” /testsettings:”MobileDevices.testsettings”


Configure the ‘Performance Test Environment’ (PTE) to run load tests in the cloud (requires VS2013  preview edition & access to the early adoption program)


NOTE: The early adoption program is limited to 15,000 virtual user minutes:


Execution is monitored in the normal way the only difference is the introduction of the Load Test Manager:


The Load Test Manager provides accesso view past load test runs or currently running load tests:


This information can also be access via ‘Team Foundation Service’ in the cloud:


The ‘Performance Test Environment’ (PTE) is now ready to be ‘Performance Test Management’ (PTM) cloud based solution on any device anywhere in the world:


Congratulations you are now ‘‘Performance Testing As A Service’ (PTaaS) Ready