Data Migrations Don’t Have to Come with Downtime

At Nextdoor we invest heavily in Redis, a highly-performant in-memory datastore, to power many of our services by providing low latency access to data. Today, we own a dozen different instances of Redis via Elasticache; many of these correspond either to our microservices or to specific use cases within our primary Django application, like AB Test Tracking. They can serve as a source of truth for data or as a pure cache for performance improvements.

With the exception of a few outliers, the majority of our Redis deployments were initially created as single instances of primary and replica nodes, so we could be certain that the failure of a single node wouldn’t cause a permanent loss of data, and also that we could continue to serve traffic in the event of a failover. Moreover, by isolating specific use cases to their own Redis instances, we could also contain different failure modes to limit the impact of individual issues. While that sounds a lot like the makings of a highly-available datastore, we realized that we weren’t quite there, especially as some of our instances reached the limits of their load capabilities. In this post we’ll explain some of these shortcomings in availability, which ultimately led us to re-architect our entire set of instances, as well as the approach we used to oversee the transition without affecting any of our core services.

Where Single Primary-Replica Configurations Fall Short

Let’s consider, for example, a scenario in which a Redis instance consisting of one primary and replica node each, and operating near max CPU capacity, triggers a failover (perhaps the replica experiences a hardware issue and a new one needs to be provisioned). The primary now begins to take a snapshot of itself to send to the new replica, which causes the primary to reach capacity. At this point, requests to this Redis instance begin to time out, and maybe some users begin to see limited functionality. We can even take this a step further, though; if our Redis client isn’t smart enough to handle these timeouts appropriately, we could cause a ripple effect with the potential to back up all traffic to our critical services.

Although this is just one specific example, we’ve already identified a set of key underlying issues that make our Redis configuration vulnerable. For one thing, CPU utilization becomes an important bottleneck in guaranteeing availability, which is never great for services which tend to be bursty and see spikes in traffic throughout the day. More important, however, is the fact that there isn’t a clean method of scaling up our Redis instances to meet the demands of our services (other than purchasing larger instance types, but this is not a very forward-thinking solution).

Fortunately, this type of issue is relatively common, and Redis itself offers a clustered mode to address it. With clustering, our data is distributed across multiple shards, and each shard owns some fraction of the entire keyspace. From an availability standpoint, this is beneficial because it allows us to scale our Redis instance horizontally when necessary (simply by adding new nodes to the cluster), without having to incur significant periods of downtime. Additionally, the impact of major issues like the one described above is mitigated somewhat; if one shard hits CPU capacity, the remaining shards can continue to operate normally, so issues are constrained to only a fraction of requests.

Devising a Migration Strategy

As we continued to scale, we began an initiative to upgrade all of our Redis deployments, with several priorities in mind:

• Switch over to clustered Redis: this would address our need for horizontal scaling and essentially solve all of the problems described above.
• Upgrade old instance types: most of our Redis instances were missing out on the performance improvements incorporated into newer hardware, so upgrading would additionally alleviate some of our concerns regarding high CPU utilization.
• Minimize downtime in the transition process: as many of our Redis instances are integral to our core services, it was important that the process of switching over to new ones wouldn’t affect the member experience.

Here we arrived at the biggest challenge: figuring out the process for the migration. Though we were very clear on what our end state needed to be, the last requirement of minimizing downtime ruled out a lot of the approaches that we might have used to finally switch over to our new instances. Notably, our goal of migrating to clustered Redis meant that we likely could not go directly through AWS, which only offers support for upgrading engine versions without changing the cluster scheme (as in, non-clustered to non-clustered or clustered to clustered). To that end, our best option would have been to manually create a new single-sharded Redis cluster, restore it from a backup of the old instance, and then scale up the number of shards later — all without being able to accept write-traffic during the several hour-long transition [1]. In order to achieve all three of our goals, we’d instead need a little bit of creativity to get us through the transition.

It was around this time that we had just started investigating the usage of Envoy, a distributed proxy built by Lyft that provides more observability into large microservice-based architectures. Envoy gave us the ability to add things like fault injection and health checking to our services for the purpose of testing our feed’s stability, simply by deploying it alongside those services. It also had the ability to serve as a Redis proxy, and more specifically to handle the routing of Redis requests across different upstream clusters [2]. Here we found the perfect abstraction we needed for a large-scale Redis migration; by adding Envoy in as a middle-man to all of our Redis instances, we could more intelligently coordinate the migration of data to our new Redis clusters while continuing to serve traffic to users. To illustrate this, let’s observe the full sequence of events required to switch over from an old instance to a new one:

Step One: Proxy all traffic to the old instance

Envoy runs as a sidecar Docker container alongside our service, and we simply modify the service to treat Envoy as its upstream Redis host. In turn, Envoy, which is listening on a specific port for Redis requests, directly forwards the requests (both read and write) to the actual old Redis instance. Importantly, other than modifying the address for Redis, no changes are needed on the client side, and the client is essentially unaware that its requests are not directly going to Redis (an exception to this occurs if the original Redis instance is already clustered; in this case, the client may need to be modified to use a normal Redis client library rather than a clustered Redis library, as Envoy behaves as a “non-clustered Redis”).

Step Two: Mirror writes to the new cluster

After provisioning a new Redis cluster with our necessary specifications, we modify Envoy’s Redis listener definition to mirror, or dual-write, to the new clustered Redis, so that any new writes go to both the old and new Redis instances. Note that Envoy adopts a fire-and-forget model for dual-writing, so it does not wait to determine whether or not the write to the new cluster is successful at all, but in general any issue here is rare. At this point, the new Redis cluster is still far behind the old one, but we have a relative sense of certainty that the two won’t begin to diverge again after the actual synchronization happens.

Step Three: Run a full backfill between clusters

This step, the most computationally intensive, entails scanning across the old Redis instance and copying all of its contents over to the new one. To do this more efficiently we use RedisShake, an excellent open-source tool from Alibaba, and wrap our own ECS service around it to automate the synchronization. Note that much like taking a snapshot, running this script has the ability to peg CPU, so special care must be taken to run the script with a reasonable QPS, or queries per second, setting, and also ideally to run it during low-traffic periods. The two instances should be perfectly in sync now, and any dual-writes that were missed have most likely been caught.

Step Four: Verify and switch over

To confirm that the two instances are in sync, we run the RedisFullCheck tool, which identifies keys that differentiate between the old instance and the new instance after several rounds of comparison and stores them in a sqlite database. In our case, key differences were more likely an indication that a particular usage of Redis somewhere was not compatible with Envoy (these incompatibilities are described below) rather than an indication of a missed write from the backfill script; some sleuthing may be required to see this step through. Once the check script does run without issue, we finally update the Envoy listener configuration again to now route all traffic entirely to the new cluster, and delete the old one.

Some Caveats

Employing the strategy above, we were successfully able to migrate all of our Redis instances to an end state where each instance was clustered and operating the latest version of Redis. In practice, however, we found that this approach was not without its limitations, especially in the usage of Envoy. Most significantly, we found that Envoy only had support for Redis requests that hashed to a single shard, which ruled out all cluster-specific commands (though since most of our instances were not previously clustered, we were fortunate to not have many of these). As a consequence, however, any transactional logic using the Redis MULTI command was also unsupported. To address this, we modified our transaction-based code to use Lua scripting instead, which offered the same atomicity benefits in a manner that was compatible with Envoy [3].

Conclusion

At Nextdoor, we’re constantly investing in improving the quality of our product and the stability of our infrastructure. Upgrading our Redis deployments was just a single piece of a several month long initiative that brought together engineers from multiple teams across the company to make our feed, one of the centerpieces of our product, something that members can rely on. These types of problems, and the brainstorming that comes with them, are what excite our engineers — as always, if they’re interesting to you too, join us!