Load & Stress testing concepts

1. Introduction

In this article we’ll present ways how to manage load testing, using the BP-Source module. We’ll also focus on performance aspects of such testing.

We have made our introduction to the BP-Source module in the previous tutorial called BP-Source: Basic testing concepts. In this article we’ll focus more at the load testing option.

There were two main reasons behind this module development. First was to use the high volume transaction generator to identify critical values of tested payment switch as stability over the time, processing bottlenecks, peak transaction response time, critical TPS value to be handled, HSM balancing, network capacity and carrying DR tests (and trainings) under a simulated production pressure. Second driver was to focus on back-office performance and quickly feeding it’s database with a large number of appropriate transactions to test reports and extracts generation times from production-size amounts of data. This test option can finally give a good answers on the switch’s hardware and software sizing and help to identify system’s braking points.

2. Test preparation

BP-Sim is designed to use the advantage of parallel processing. This is achieved by running each terminal (ATM, gateway) as a separate thread. Such configuration will appear to the tested environment as completely independent connections from multiple devices. So at the first stage we need to tell BP-Sim which terminals will be used for transacting and how many transactions should be carried. Now use the BP-Sim console and navigate to terminals configuration. As you may see on the screenshot below. Each terminal allows you to set it Active for processing – see a checkbox next to the terminal id. The same can be achieved through the context menu called on the terminal name within the BP-Sim navigation tree. Many content management things can be achieved through this context menu and our clients are encouraged to use it.

Note that there are already 4 terminals activated for processing on the screen shot below. This can be realized through the the leading start “*” at the very beginning of the terminal’s name in the navigation tree. We have also set this terminal to send 10k transactions, by using random selection from available cards. Very interesting option is setting up the amount increase value for your transactions. This very simple feature allows increasing the transaction amount by the value given. So every following transaction will have it’s amount higher than the previous one. BP-Source application will continue raising this value to 999.00 before reseting to zero and starting again. There is no test script assigned to this terminal as this is a subject to our next tutorial, but this option brings further interesting options which will be covered later. Our terminal, as currently configured, will also wait 2000ms for a response from a host and will keep persistent connection with it (e.g. not establishing & dropping a connection for each transaction, just one network socket will be used during the whole test run).

Last important part is not forgetting to push the “Save” button for storing your current terminal configuration into application database .

Last step ion our configuration is setting up the source itself. This is being dealt through the Sources screen, where we set testing mode to “Load testing” and “Transaction load” (TPS) value. TPS here means a maximum number of transactions generated during a period of one second from all transaction sources in total. Note that a real number of transactions carried may rely on many factors. As the first, imagine having a terminal configured to wait for a host response for 2 seconds. It is clear that configured TPS value cannot be achieved in case that our host stops responding in a some usual manner. Another limiting factor is TPS level license available. While any TPS value can be configured, application will always take license’s TPS as the upper limit. Easy way how to set maximum processing level is setting this value to zero “0”, which will tell BP-Sim to setup maximum available TPS value on your behalf.

3. Running the load test

Now when we have configuration done, we should think about what is going to happen during the test itself. It is good manner trying whether the tested host is visible to our test tool and it the  BP-Sim and also tested host has appropriate resources available for our testing. Usual way to check host availability is by attempting a simple telnet connection to desired host address and processing port. Regarding the second thought, it is always wise checking main processes running in the background and see if there might be any application interfering with the test run as database extracts, disk backup or some resource intensive (hdd I/O, CPU, RAM) operations in general.

We can be dealing with pretty high transaction volumes here so we should also justify whether the test should be whether our host is set to do the on-us authorization, or our transaction stream will be send to the upstream entity. In case of latter, we should be always sure that we are not going to create some issues to somebody else as there might be shared network segments between local testing environment and production, while e.g. forwarding your transaction stream for further processing outside your company network. In these case it is always better to choose local authorization, where you may almost always choose between switching to stand-in processing or having an upstream simulator running – as BP-Host.

Our last check should be reading the settings overview tab located at Source control screen. On the screenshot below you can see that our test should carry with 4 terminals, having each of them set to send 10k transactions. Lets launch the test. The log window should provide you with information on current processing, where each transaction is logged with appropriate time-stamp.

Adding also corresponding BP-Host screen, having similar processing log there as well.

4. Test results analysis

Our test is over, now we need to settle with what was achieved. Screen below shows brief test statistics reported on test end. On example result given we can see that each terminal was processing with average 4 TPS, getting cumulative 16TPS. Very important information is located within the #Pass/Fail column, showing that more than 2/3 of transactions were not responded by the host at all. It is important to note here that this value ratio is given on the transport (network) layer. So knowing that certain number of transactions were responded still doesn’t exactly mean that those were approved or processed correctly.

BP-Sim test output gives us several options for the test analytics. In our scenario we have complete traces for all transactions carried, we also have the log output from the BP-Source having general test statistic there and finally we should have full-detailed trace from BP-Host, behaving as our authorization entity. Now it should be very clear to find breaking point of your system, whether it was processing application, database, network, issue needs to be isolated and problem fixed.

Note that such performance testing might be assisted by running monitoring applications on the tested system as perfmon for MS Windows, or e.g. top on Linux, but these are out of scope of this tutorial.

Just for example the screen below shows a BP-Source reporting a socket operation failure, while attempting to deliver data to a network socket. This should give a very good clue to any networking/application engineer where to have a look.

5. Possible performance issues

BP-Source and BP-Host applications are written in a way to utilize your desktop resources in the most effective way. This simply means that if you are having an average office desktop or laptop and testing with TPS below 1000, you should not encounter any performance issues during the test run. We will dedicate another tutorial to load testing with TPS over 1000 as hardware and OS needs be considered for such testing.

There are some ideas in case of any doubts regarding BP-Source & BP-Host performance:

  1. Check that your testing environment has more than 2 CPUs available, when running BP-Source & BP-Host in parallel from one desktop.
  2. Check that your testing environment has more than 1GB RAM available.
  3. Check that your testing environment has at least 1 GB disk space left for writing traces and advanced I/O functionalities (SMART, caching) is enabled.
  4. Reboot your environment to release resources taken by other processes.

Note that there was no performance impact observed while having BP-Source or BP-Host tracing enabled/disabled. Also there was just minor impact noted while using DUKPT security for full message MACing and PIN encryption, so it is recommended to let all these parameters set to bring your test run as close as possible to the real world scenario.

6. Load testing checklist and testing suggestions

We have prepared following check-list to aid your load testing:

  1. Have you set desired number of active terminals for processing?
  2. Have you considered reseting transaction counters?
  3. Have you moved & backed up BP-Source & BP-Host traces (if enabled)?
  4. Does all active terminals have reasonable number transactions to be send?
  5. Do you want to send transactions from one card or from random selection from all cards marked as active?
  6. Did you assigned appropriate TPS level in Source’s configuration?
  7. Have you checked host’s network availability?
  8. Are you running any data analysis supporting applications (perfmon/netstat/wireshark/iotop/host traces)?
  9. Is your authorization service ready?
  10. Is your upstream authorization service ready?
  11. Have you checked “Current settings overview” and validated that test setting is correct?
  12. Are you sure that IP address and port configured is not taken from production system?
  13. Have you considered all possible threads, related with your testing, to the production system (e.g. testing system located on the same VM-ware)?
  14. Launch the test.

Further suggestions for load testing:

  1. Try to disable “Connection persistence” parameter for terminals, letting BP-Source to open a new network connection for each transaction.
  2. Enable and disable security settings to reveal possible HSM performance and  balancing issues.
  3. Prepare several load testing runs and increase TPS each time to reveal your authorization service breaking point.
  4. Use constant transaction load for DR situation simulation and measure how much transactions were dropped out during the outage.
  5. Train your staff for DR situations, under the pressure of a simulated real volume processing.
  6. Benchmark your system for the peak values.
  7. Use BP-Source for your environment sizing: Fill your testing database for your processing server as well as back-office server to the design levels and try running clearing and settlement operations over those.
  8. Try full database backup, while processing at peak levels.
  9. Refresh your switch configuration, while processing at peak levels.
  10. Simulate primary HSM drop-out  and let all security taken care of by HSM located at your DR location.

7. Summary

In this tutorial, we did an introduction into the Load testing mode for BP-Sim’s Source module and covered most important parts of it’s settings. This article also provided clues on how to analyze load test results and address some possible performance issues. Finally we did provide load-test checklist and some suggestions on testing options.