Speed, Security, and Scalability: Pick Only 2!

This article will break down this question into three attributes: Speed, Security, and Scalability (and call them the 3S).

First, we will describe some of the challenges engineering teams face when they simultaneously try to be great at the 3S.

Next, we will review how companies of all sizes might change their strategy to adapt to their business needs. And finally, we will look at a few interesting trends that could help some engineering teams in their journey.

What is 3S?

Speed: The time it takes for an idea to go to market

Metrics: Time to market, number of features shipped.

Traits observed:

• Individual contributors have a lot of autonomy and can move fast
• Fewer processes and checks when making changes
• Innovation is a differentiator.
• It can lead to feature creep and tech debt.

Security: The process of incorporating security controls into the information system

Metrics: Number of security incidents, weighted by severity, Compliance achievements.

Traits observed:

• Individual contributors have shared ownership and keep each other accountable
• Many processes and checks when making changes
• Innovation could be seen as a risk
• Compliance and security requirements become key projects, taking over the product roadmap at times

Scalability: The ability of a system to change in size or volume

Metrics: Performance and Uptime metrics

Traits observed:

• Individual contributors rely on tools and automation. Setting up this automation can be complex and time-consuming in the initial stages
• Increased automation
• Infrastructure as code
• Multi-region, Multi-cloud
• Tech debt can impair scalability

Why trying to excel at the 3S can be counterproductive?

While these three attributes are well-intentioned, they might sometimes act as opposing forces. Let’s provide a few examples.

Speed working against Security: To prevent developers from being blocked, one might give them full access to some systems. But that would also go against the security requirements.

Scalability against Speed: One scalability goal could be to use multiple regions, possibly multi-cloud providers. But that would make the release and deployment process much more complicated for developers. This also creates a new problem; the application data is now stored in many places.

Security against Scalability: To improve security, every new system or library might require an approval process before using it. But that could mean introducing a new tool or technology to improve scalability (ex: Kubernetes) is slower. Furthermore, it can create deadlocks where innovation is a risk in the worst case.

We could provide more arguments on where to put the cursor and lead to emotional discussions for all of these examples. The answer might be different for each company. But the point is made, these can be opposing forces. If a business has limited resources, it might want to make an educated decision about where it would like to put its focus to reduce these friction points.

Introducing unnecessary complexity when the business doesn’t need it can be damaging. Maybe the current team does not have all the skills required, so they will need to ramp up. It might slow down the ability to go to market. Some teams would work on some projects that turned out to be not required. Having opposite forces might lead to heated discussions and delay decisions. Ultimately it could negatively impact morale and lead to attrition.

We can parallel the CAP theorem and state that an engineering team cannot simultaneously excel at Speed, Scalability, and Security to push this even further.

The CAP theorem states that a distributed system cannot simultaneously be Consistent, Available, and Partition tolerant.

Big disclaimer: it’s not about dropping one attribute but accepting that the focus should be on the two other attributes at the current stage. The engineering team will be more focused by clearly stating what the company needs now and communicating about the accepted tradeoffs. A great leader would anticipate a step further in their message and explain how they would address today’s tradeoff in the future. And prepare accordingly when the time approaches.

Which 3S to pick?

Early-stage startups value Speed before anything else. They want to find their market and keep their customers happy by shipping new product features at a high pace.

Some industries such as Finance or Healthcare might dictate that Security alongside Speed is the top attribute to focus on.

More prominent startups experiencing high growth would usually favor Scalability and Speed. If they compete in a winner-takes-all market, that strategy will allow them to execute on their product roadmap and reach out to new markets faster. If they are preparing for IPO or expanding globally, their compliance requirements will make them pivot to focus on Security and Scalability.

Finally, big tech companies with years of experience had time to learn and went through the many stages themselves. As a result, they have some of the best and largest engineering teams; they excel at the 3S.

This is also not uncommon for large companies to spin up a unique project into an independent entity to enable it to move faster, sometimes also referred to as an Internal Startup.

What’s next for the 3S?

Nobody can predict the future, but we can observe some trends in the market and report some of the feedback we hear from our users.

Speed: hiring is the bottleneck, a new wave of startups coming in

Early-stage funding is at an all-time high, and it’s a race for these startups to find their market. The number one impairment to speed this day seems to be tech hiring. Because hiring is slow, they want to do more with fewer people. A new wave of PaaS and software simplifies everyday engineering tasks.

Simplifying cloud computing is our mission for Opta, but there are many other new startups whose goal is to help developers. We have a few exciting examples among our users: Canvas, which brings SQL to non-technical teams, and Union.ai, which simplifies machine learning.

Scalability: Kubernetes adoption, multi-cloud, and demand for opinionated frameworks

There is a big industry trend to move to Kubernetes. But it comes with some challenges and frequent upgrades (not all backward compatible). Using Kubernetes is a journey, not a destination. This complexity is sometimes misunderstood and can negatively impact the product roadmap by holding engineering resources.

The latest HashiCorp State of Cloud survey reported that multi-cloud is now the new normal. Cloud providers are very different, and supporting many will create much more work for the engineering team. Our customers want Kubernetes on their cloud provider but don’t necessarily have a strong opinion on how they should configure it. They want best practices, such as application monitoring and Infrastructure-as-Code.

Security: help needed with Kubernetes networking and IAM

Here again, moving to Kubernetes creates more complexity. Most cloud providers provide a service to run a cluster, but it’s up to the user to set their network (ingress, service mesh) and IAM for their cluster. Some companies start with Kubernetes; they don’t want to start from a blank state. They appreciate using products that can provide an opinionated way to configure their networking stack, from their Load Balancer to the application pods running on their cluster.

What we have seen

This article looked at engineering efficiency by considering the 3S: Speed, Security, and Scalability. For many companies, trying to excel on all 3S too early can create unnecessary complexity, increasing the tech debt. This could ultimately create frustration for the engineering team and lead to attrition.

Depending on the stage and the market where the business operates, it might make sense to focus on one or two attributes simultaneously. Communicating clearly about the current strategy and the tradeoffs will help the engineering team stay focused.

The latest trends, such as the tech hiring crisis and the popularity of Kubernetes, have left some companies turning to new products which simplify their engineering tasks so they can execute on their roadmap with fewer hires.

How to Manage your Team’s Infrastructure Easily

Opta, our free, open-source tool, reduces most of the complexity of cloud infrastructure and pipelines for your team. It is an Infrastructure-as-Code framework built on top of Terraform that lets you work with high-level constructs to set up custom cloud infrastructure quickly.

Want to Connect?If you run into any issues in the installation stage, you can reach out to the contributors in our slack channel here