How Zerocode helped testing Disaster Recovery Solution for AWS MSK ?

— Events based on True Story ;)

Photo by NOAA on Unsplash

Disaster & Recovery

Consider a scenario where AWS Managed Streaming for Kafka (MSK) hosted in an AWS region. If the AWS region goes down or the MSK in the region is not available, the applications consuming MSK would be blocked. There comes a simple Disaster Recovery (DR) solution.

Simple Disaster Recovery Solution to be Validated.

For the sake of simplicity, let’s assume:

• The MSK cluster is in Primary region.
• The backup MSK cluster is in Disaster Recovery region.
• There is an active data replicator replicating the Topics, Messages, Offsets and what not from Primary MSK to DR MSK clusters needed for Business continuity.
• The applications (Message Producers and Consumers) are being run from AWS Elastic Kubernetes Service (EKS) Clusters as Pods.

I need to validate the Functional aspect of the infra:

• MSK clusters managed across 2 AWS regions. So the test need to connect to both the clusters
• MSK was using SASL/ SCRAM for authentication.
• SASL credentials hosted in AWS Secrets Manager.
• Test same way as producers and consumer which were dockerized and hosted in EKS Clusters.

Too much of AWS components !!! Hmm.. Now the Zerocode’s play…

Bit about Zerocode…

Zerocode Open Source is a lightweight, simple and extensible framework for writing test intentions in a simple JSON or YAML format that facilitates both declarative configuration and automation. Further read @ https://github.com/authorjapps/zerocode

Basically the test can be written as JSON/ YAML intentions. Additional overhead of implementing Kafka API calls are handled effectively by Zerocode. A simple test case to produce and consume a message would look like:

Ref: Kafka-Testing-Introduction

Journey of unravelling issues using Zerocode…

Issue #1: Connecting to Multiple Cluster in a single Test

For the evaluation, I needed to connect to Primary cluster (as message producer) and to the DR clusters (as message consumer) within a single test. This is to evaluate the Topic state, Message’s flow & idempotency through the data replicator.

Zerocode provides Custom Kafka Client creation to override the default Kafka client behaviour for the test needs and we can override the send and receive mechanisms effectively.

Screen shot from zerocode custom Kafka Client example.

Issue #2: Handling SASL SCRAM Auth.

The auth credentials for MSK are retrieved from Secrets manager by the deployment tool but to pass it to the Test execution environment there needed to be a mechanism.

Zero code provides reading the System Properties and System Environment variables. This retrieval call be invoked in the Test intentions JSON config with ease.

Reading System Properties and Environment Variables in Zerocode Test Steps

Issue #3: Reading dynamic values from previous step execution.

Creating new messages every-time for test will have new offsets. So, validating the messages created in current test cycle was an overhead.

But Zerocode provides option to read the response of the produce step and re-use the offset value in the consume step. This avoided the overhead to create clean-up the Topic or create new Topic for every test.

Example implementation in HTTP based test. Ref: Query

Issue #4: Invoking Custom Java Methods:

There will be situations to invoke Java Utility methods in the middle of testing. Zerocode even added option to invoke the Custom java methods as test steps.

Issue #5: Load Testing

Zerocode supports Performance Testing. Detailed read: Load-or-Performance-Testing-(IDE-based)

Issue #6: Dokerizing the application

Being a java based implementation, dockerising the Zerocode based test implementation was straightforward. I compiled the entire application into an Uber Jar (‘Fat Jar or Jar with dependencies) using the Gradle.

Ref: Gradle: java_packaging

Then loading them in a JDK based docker image (e.g. adoptopenjdk/openjdk11) made it easy to dockerise the test solution.

So, I’m now be able to implement a one-stop solution for testing the functionality & Performance of the MSK DR infrastructure using Zerocode.

The Metrics Monitoring, Alarms, Failover/ Failback strategy and likewise which are key for DR implementation, were taken care outside of the Functional testing scope.

Final words.. Not Mine.. But by Zerocode though !!!

The bottom line is that Zerocode Open Source makes automated API testing declarative and simple. If you’d like to learn more, visit the quick-start guide or user’s guide to get started testing — fast!

Happy testing!