What is Something We All Have But Don’t Own and Never Works When We Need It? Test Enviornments.

Overview

Let’s first get the elephant out of the room. You came in here thinking it would be about dogs. It won’t be about dogs. But I will mention them because I like dogs (just not when they are barking).

We all know that testing your jobs and applications is important as we want to ensure a stable and seamless customer experience, as is expected by consumers nowadays. But more often than not, things are just not quite working as you need in your test environments. One team has pushed their latest changes which are incompatible with yours. The data in this environment is not good enough. The test environment is down because of a resource crunch or an infrastructure change. We can’t use this environment because it’s dedicated to load testing for the next 2 days.

Now we are called for a status update where we have to explain our problem. Most people resort to the word “blocked”, which I hate as you are essentially saying you have exhausted ALL possible options and done everything you possibly could and still can't move forward. It tells me that you have given up, want to chill a bit and wait for others to solve it. These are the situations that define team players vs individuals.

Okay, rant over. So what are we doing with our test environments that makes it so difficult?

Relationship status: It’s complicated

Exactly. Just like my relationship with my dog. The slightest change in the surrounding environment can cause an episode of non-stop barking for 5 minutes, just like when a test environment gets updated with a service with broken code. You need to tend to and care for it, feeding it with food and walking it every day. The dog owner is responsible for this but who is responsible for the test environment? Is there a test environment owner?

Generally speaking, no. It is a shared responsibility. Therein lies the problem. Shared responsibilities require communication between all the parties involved. And from my experience, more issues occur wherever more communication is involved. Each day, you always ask “Have you fed the dog?” or “Has the dog gone for a walk?” but what questions do you ask of your test environment? Do you ask all your dependencies about what they are doing in the test environment each day? No. We make assumptions that everything should be working. And if it isn’t, we are usually quick to blame (I will refrain from my “blocked” style rant again here).

As problem solvers, we know that there is a need for a tool or process to try to resolve this problem.

Tools in the shed

So then each team tries to go about testing their little part of the world. They discover tools that can help execute them or they make use of their software development skills to create an in-house solution that fits the exact needs. “We use Postman to test out the account APIs”. “Our bash scripts send messages to the Kafka topic”. “We created a scalable, high-availability, consistent job to create dummy records in Postgres”. Soon enough, you will have the “too many tools in the shed” problem. This is where you have accumulated enough “tools” that consume all your time to maintain, upgrade or choose which tool to use, such that it outweighs the time spent on actually trying to solve the original problem.

So let’s take a step back and understand what value we can gain from having a test environment before we jump to the conclusion of saying it is too difficult.

The VALUE of a test environment

This depends on the specific context of your business and tech stack, but reasons why having a test environment is important include:

• Test integration between tools and services: Run and execute jobs/services connecting to upstream and downstream systems.
• Chance to catch issues before production: Stop costly bugs from reaching production, less time to resolve production issues.
• Increased stability and reliability: Customers experience what others have already tested.
• Ability to experiment: New technologies or solutions can be tested to validate assumptions, improve processes or help customers.

The PAIN of a test environment

Although it’s always easiest to point out problems, we will summarise the main pain points below. These are:

• Additional cost and maintenance: Infrastructure, data storage, jobs and services all need to be maintained and updated.
• Difficulty in replicating production: Complex relationships between systems, networking, resource allocation, data quality and size all add up to make it difficult to replicate production.
• Time constraints: If you have tight deadlines or need to fix bugs in production as soon as possible, not having a test environment will help you ship faster.
• Simple problem, useless testing: Simply put, the problem you are solving is not complex and your solution reflects that. Unit tests are sufficient and you do not require integration tests.

Conclusion

Understanding the value of having a test environment is important to keep our users happy but we must consider the cost. We could try to solve the problem by throwing more resources at it. More testers, more steps in the process, more tools in the shed. But when the shed gets full, you have to change the way you think and find other ways to solve it without filling up the shed. We therefore shift our focus on solving or easing the pain points, trying to aim for unified tools or processes that developers, testers or even business analysts could use to ensure our jobs and services will work as expected.

**Check out Data Caterer to see how it can be used as a unified tool to ease the pain of cost, difficulty in replicating production and time constraints with automated data generation, validation and cleanup.