Centralised Testing Functions – Oversight and Governance

I’ve been working on a project over in Sweden, and have found it unlike anything I’ve ever seen before. Sadly, this is for all too familiar reasons.

I work for an organisation with a small performance testing function centralised across all the projects that can be conjured up. As a result, there are often a lack of resources available to conduct testing, or even to conduct an assessment of whether testing is necessary and appropriate. The knock-on effect of this is that large third party consultants are drafted in to pick up the bigger projects and the centralised testing function – the Test Center – take a back seat.

In practice, that back seat is often in a different vehicle, and so the Organisation as a whole lose ownership and governance of their own IT projects, ceding it to the consultants.

It is in the nature of the larger consultancies to be concerned with one thing only – Successful Delivery. It is a known side effect of this concern that the contracts will often be drawn up to include bonuses and interim payments based on successful delivery.

In many ways this is absolutely fine, but it requires an oversight position from the test center to conduct reviews upon such deliverables as:

  • the test approach
  • the test scripts
  • the test results
  • defects
  • the deployment decision

The question is: What happens when you have an immature IT organisation, who have ceded control of the project to a large 3rd party supplier – focused on delivery at all costs – with no checks and balances in place to ensure the quality of that delivery.

We end up with an immature IT organisation, an obfuscating 3rd party vendor hiding meaningful results whilst ensuring that something (ANYTHING) is delivered, so that they can meet the project milestones and ensure payments are made.

Essentially, the organisation order 1 ton of oranges, the third party vendor deliver 1 ton of lemons and claim success.

Importantly it is in no small way the fault of the purchaser for ceding that control in the first instance.

Let’s be clear, the larger third party suppliers often use primarily low-cost offshore resources. These resources are typically low-cost due to a lack of experience and exposure to the larger business-world around them. Additionally, these resources are used as a delivery factory, churning out scripts of varying need and quality according to a vague set of requirements. The requirements are generally vague due to the IT immaturity of the organisation.

The 3rd party vendor then adds a layer of project management around release and defect management whose role seems to be to obfuscate clear communication, to actively seek to remove any review process and oversight from the organisation. They effectively become closed shops of development and testing who produce results that mean very little to the overall organisation but ensure that project milestones are met.

Everything becomes deadline-based, and measured in time rather than overall quality.

All of which can be avoided by maintaining a level of governance and oversight over the project.

The delivery factory will continue to churn out scripts of varying quality and need but this need and quality will be assessed and reviewed by an employee of the test center. The same applies to the approach documentation, the test scenarios, the load and data models.

Python Functions

Declaration of functions in Python is nice and simple.

def my_function():
print("Hello From My Function!")

The call is also uncomplicated


With Arguments

def sum_two_numbers(a, b):
return a + b


Python Basics

The absolute basics:


print("Goodbye, World!")
x = "string value"
print (x)


myint = 7


myfloat = 7.0
myfloat = float(7)

Extending Loadrunner Scripts with C – Function Library #1.2

I have now left the Cardiff JMeter role, and returned to Loadrunner work, this time in Dublin.
With every new position I find myself re-invigorated to learn new things, update old code and write some articles for the site.
I have a lot of ideas in my head at the moment, but my shower thoughts this morning were to finally transform the writeToOutputFile function into exactly that, a function.

Jumping Ahead with JMeter

A new article in the dev diary describes how I’ve gone about creating a fully automated performance test suite using jmeter for my latest clients. A further article will shortly appear here, although my concept of shortly may be somewhat different to most.

In any case I have plans to document the process of constructing JMeter test plans and fragments and then to document how to extend that into a full suite of tests, and then how to execute that without a lot of manual intervention, and potentially, I may revisit running that without ANY manual intervention So there’s that to look forward to in 2015, hopefully in Q1.

String Manipulation with VBA

As a performance tester, most data arrives at me in a spreadsheet, or an enormous text file with little or no delimitation.

The previous article showed how to parse across columns and down rows to gather data.

Excel has a number of built in functions to manipulate that data before outputting it for us.

Functions like

MID To pull a child string from a parent string specificied by the number and position of characters
LEFT / RIGHT Similar to mid, but from either the 1st or Last character in the string
TRIM Removes whitespace from the ends of strings (both ends)

These help immeasurably in cleaning up the data.

Tied into further functions like

We have the facility to parse through raw data, picking up the elements we need and outputting them in the form we require. Given that loadrunner uses .dat or .csv files for test data input, this is hugely useful.

Instr is specifically used within VBA, the other functions mentioned can also be used at the worksheet level and embedded into formulae.

Instr Search inside one string for another

Excel as an IDE

I’ve often found, especially when starting at a new client, that real development tools aren’t provided for testers (even automation testers) as standard and have to be requested from the helpdesk. This can take a week or longer to sort out, depending on efficiency. It’s worse in financial institutions who seem to think Admin rights are golddust, who lock down online storage and usb ports, who scan and validate every email. That’s their prerogative, but if you’re going to pay me to do my job, maybe make it so I actually can rather than putting security in the way. A lot of it is foolish, I ask for read access to a customer database in a test environment. Now as long as you’re not using live customer data, there’s virtually no risk to granting that, it’s a 5-minute job – so why does it alway’s take a week and 3 managers to approve it?

In any case, my workaround for this is MS Excel.

There are a number of reasons for this:

  • Because of the structure of a workbook, and a worksheet, into tabs, rows and columns, it’s easy to visualise data structures like lists and arrays.
  • Because rows and columns provide an easy structure to navigate around in code
  • Because I have used Visual Basic for a long time, and Excel’s VBA is probably the easiest variant of this to use
  • Because every client I’ve ever been to has Excel.
  • Because you don’t need admin rights to run macros’ typically.

Performance Testing – 4 Questions (an approach)

I’ve been asked to give a talk to some of our off-shore developers. They’re sitting through a workshop on just about everything related to the client environment, and I’m talking for 15 minutes about performance testing with Loadrunner against web-baed sales flows.

I thought about winging it, but the manager has asked for a powerpoint deck, so I’m assembling that now. And as I get to the first page about entry criteria and intake activities, it occurs to me that I’ve not defined this before although it’s always in my head and is a very common interview question and answer.

I had a chat with my colleague and explained to him that I use 4 questions to gather all the information I need to build and execute a performance test. He didn’t believe me and say’s he uses nine questions to get that information. So we discussed it some more and reached an understanding.

I’m talking about the 4 top-level first questions to which I need an answer before I’m building anything. Other questions will be required as we move forwards with the project. And the answers to the 4 questions will evoke other questions and answers but at the highest level, I walk into that first client meeting armed with my 4 questions.

  1. How is it built? – The solution under test
  2. What request(s)? – the script(s)
  3. How many? – requests per hour – the scenario
  4. How quickly? – do they need to go? – the requirement(s)

MM (my colleague) wants to ask about network topology, about transactional drop-out rates, about reporting structure and team contacts etc. I argue that isn’t necessary in the first instance, but is useful to know as and when, or before the first execution.

Simply, those 4 questions define the solution, the scripts, the scenario and the pass/fail criteria. From that I can build the test. Maybe it’s experience, maybe I’m overstating my abilities, or maybe I’m wrong, all I know is that for 15 years, I’ve been a performance tester and my refined approach begins with those 4 questions.

Loadrunner misses extra resources

Recently, I’ve encountered an issue with an AUT where it crashes VUgen during the generation stage of a recording session. Now I’m a big fan of record-edit-playback. I find the recording and generation logs hugely useful and I find that recording is the easiest and quickest way to get a view of the application.
I call it record-edit-playback because the recording isn’t enough on it’s own for me to use as a performing performance test. But as a foundation, it’s fine. And they rarely playback immediately in any case. Session ID’s are designed to prevent that kind of thing.

The logs in particular are great for helping you find and construct parameter-capturing code for session ids, and identifying user variables that you may want to emulate.

My process for recording is very simple though I haven’t documented it here before so I will now:

  • Record the script once
  • Save it as Scriptname_DATE_RAW
  • Save it again as Scriptname_DATE_WIP (work in progress).

That means you can always roll back if you need to. The WIP copy is the working document until it’s finished when it becomes Scriptname_DATE_FINAL. At that point I hive it off to the controller as well as my local repository. I don’t like cluttering up the controller with WIP versions. And I don’t like the controller pulling scripts across the network, I just think it’s poor practice.

But I digress.

As a solution to the fact I couldn’t record the script, I used Fiddler as a proxy to capture the urls I visited when manually executing the script in Firefox. Over on the DevDiary there’s an article about this, but the point I wanted to make was this. Loadrunner doesn’t capture everything that’s on a page. Fiddler output was about 40 lines for the homepage, a Loadrunner visit to just the homepage was capturing 10 lines of resources (I managed to get LR to do that before it died again).

It seems that if a resource (for example a .css) contains sub-resources Fiddler will see that but Loadrunner won’t. I don’t know if that is by accident or design. I don’t know if it’s implied that LR is geettng them but not explicit in the results and the logs. I intend to find out in due course but it makes me wonder how I’ve not seen this before in 15 years of performance testing. Maybe it’s specific to this project, I could believe that, we are uniquely complicated from what I’ve seen. But what if it’s not. How many issues could have been avoided if I’d seen a bottleneck on one of those resources – maybe an underperforming java-script for example? It’s all academic now anyway but it’s certainly something I’ll look out for in the future. And as an aside, maybe Loadrunners recording engine isn’t as good as I’ve always thought it to be? Interesting times. In Belgium…

Dev Diary

There’s a link to the dev diary over there on the right, currently amongst the links to hosts and wordpress themes. I’ll be adding more relevant content as I find it.

I’ve added a few days of diary regarding test stubs, vbs and vb6. It’s coming along nicely.

The dev diary will be much more informal that the main AS.Org site, this one is going to remain as my more professional aspect.


The first working version of the Loadrunner Batch Schedule (Excel version) is now available on Dev.

Powered by WordPress and ThemeMag