author avatar
By Ephraim IsrailyQA Engineer

*Views, thoughts, and opinions expressed in this post belong solely to the author, and not necessarily to SemanticBits.

So, your project team has QA’d the heck out of the newly developed service, both manually and with a slick automation framework. Your confidence level is very high and you’re ready to ship. DevOps has a very smooth release, QA passed all PROD smoke tests, and you’re celebrating. 

Suddenly, the field reports complaints of blank screens, underlying 500 errors or indefinite, cute-looking spinners. And, yet, nothing slipped past DEV unit tests, QA, or Product Managers, and it works in all environments.

Anyone with experience going through full 360° SDLCs, will have encountered performance issues at one point or another. As a software development shop matures, it realizes the fundamental importance of performance/load testing and allocating budget and time to this initiative. You may have created the coolest bleeding-edge technology with wonderful integrated layers, and may be considered the Shakespeare of bits and bytes. However, from a business perspective, if the service doesn’t respond to tight, precious time, it’s inefficient and counterproductive. A business person doesn’t value, look at, or even want to comprehend the intricate underlying masterpiece.

Historically, some of the reasons for neglecting performance testing were due to the enormous costs of the tools and hardware platforms needed for this non-functional requirement. I remember back in 1999—during the dot-com bubble era with deep-pocketed Wall Street investments pouring in—when licenses for Silk Performer were purchased for 6 figures! Another reason is, ideally, you need a staging replica of the production environment, a very costly addition indeed. Where I worked then, we had a million-dollar fridge-like EMC box just for staging. Those tech golden years are way gone. So, what now?

With the advent of open source and AWS, things have changed drastically for performance testing. Now, even small shops are up there with the big gorillas. The most popular and mature open-source performance/load testing tool is Apache JMeter, with a large support community and a plethora of plugins. It has a fairly small learning curve, can run headless and integrate with your CI/CD pipeline. The authoring of performance/load tests are very similar to that of Apache NiFi, with a frontend GUI saved to XML as a JMX file.

So what is performance testing and how do you go about it with your project? 

No need to compete with this definition, so let’s jump right in with the how.

How can you replace real world users, on a wide range of different computers or devices, all hitting the service at the same time? It would be farfetched to launch an automated test on 1000 machines. This would incur astronomical costs together with astronomical maintenance nightmares. 

The answer is you can’t. You can only simulate the backend calls their browsers are doing or what is called XHR (XMLHttpRequest). (On macOS -> chrome -> type ⌥⌘i -> DevTools -> under the Network tab -> records of Type xhr are visible).

The first step is for Product to assess the expected user base, which will concurrently hit the service you’re developing. If the number is less than ~250 concurrent users, you can use JMeter and run one instance of the tool to easily simulate the 250 concurrent API (XHR) calls for a scheduled load time. The tricky part is when you need to run thousands of concurrent users. For this effort, multiple instances of the tool are required to be invoked in a coordinated, distributed way.

What is JMeter distributed load testing?

If we are running load tests locally with JMeter, there are certain limitations to the number of users you can run, even if your computer has enough CPU and memory.

How can we create a scenario with thousands of concurrent users using JMeter? One of the answers is running JMeter in distributed mode.

For those of you who have never heard about it, here is a brief explanation.

When we talk about distributing JMeter, we refer to a Client-Server architecture where JMeter uses Java Remote Method Invocation (RMI) to interact with objects in a distributed network.

You can see this in the images below:

Docker style:

Distributed testing enables JMeter (client controller) to handle the test execution, together with multiple remote JMeter instances (servers or workers) that will send the request to our target test application server.

All resulting Key Performance Indicators (KPI) metrics are sent back to the client JMeter container to aggregate and are eventually built into reports and/or charts for analysis.

Taking JMeter distributed load testing to the next level with Kubernetes (k8s)

When faced with the challenge of where to find all these machines to run the distributed load tests, Scott Stuart of QPP A&R DevOps at SemanticBits, with his keen k8s and AWS insights, advised me to look into JMeter on Kubernetes.

At first, there were some challenges with getting the Docker images (client & server) to the correct state. But after a couple of iterations, the framework turned out amazing. For each test where scaling is needed to run the amount of required concurrent virtual users (VU), all you need is a simple command:

kubectl scale deployment -n load jmeter-servers --replicas=4 

With the above k8s scaling capabilities, the JMeter server Pod is replicated 4 times to run 1000 concurrent VUs. No maintenance, no configurations, no need for fresh new virtual machines. You can create, then destroy, the Pods after the test is done with:

kubectl scale deployment -n load jmeter-servers --replicas=0 

Here is a sample of triggering JMeter headless for a schedule of 250 VUs on a distributed k8s platform:

jmeter \
  -n -t $JMX_FILE \
  -LWARN \
  -G user=$USER \
  -G password=$MY_SECRET_PASS \
  -LINFO \
  -G env=qa \
  -G timer=500 \
  -G "threads_schedule= \
  spawn(20,1s,200s,200s,459s) \
  spawn(20,1s,300s,200s,459s) \
  spawn(20,1s,400s,400s,459s) \
  spawn(20,1s,400s,400s,459s) \
  spawn(20,1s,400s,400s,459s) \
  spawn(20,1s,400s,400s,459s) \
  spawn(20,1s,400s,400s,459s) \
  spawn(20,1s,400s,400s,459s) \
  spawn(20,1s,400s,400s,459s) \
  spawn(20,1s,400s,400s,459s) \
  spawn(20,1s,400s,400s,459s) \
  spawn(20,1s,400s,400s,459s) \
  spawn(10,1s,400s,400s,459s)" \
  -l results_load.jtl \
  -Dserver.rmi.ssl.disable=true \
  -R `getent ahostsv4 
'jmeter-operator-loadtest-jmeter-serv-svc'  | cut -d' ' 
-f1 | sort -u | awk -v ORS=, '{print $1}' | sed 's/,$//'`

 A graphic representation of the above VU startup, hold, and shutdown schedule:

So, if your team is developing microservices running on Kubernetes, and needs to test the k8s horizontal Pod Autoscaler, this load test framework will come in quite handy.

Your target server under heavy load test (Neo) will be challenged against replicated cloned JMeter server Pods (Agent Smith).

⚠️⚠️⚠️ be ready for the fight ⚠️⚠️⚠️