A Blazing Fast ️⚡️ Application Layer Cache for Elasticsearch

Appbase.io Cache is an application layer cache for Elasticsearch designed for the most demanding application search use-cases: whether you’re performing a search query, applying a filter condition, or running aggregations — our cache just works. It offers a high-degree of configurability and also comes with a point-and-click control plane.

If your use-case is application search, appbase.io’s cache can save up to 100ms in search latencies while simultaneously drastically improving the search throughput. If 80% of the search query volume in your application is made up of repeat queries, then the throughput improvement (translates to cost savings) that you will see will be 4x.

How does a 100ms latency save translate to dollar value? In a now popular study, Amazon.com found that they would lose a billion dollars in annual sales with a 100ms drop in their application search latencies.

These are bold claims! We will break down the why and how behind them in this post.

Why can’t Elasticsearch itself cache

The most obvious question that might come to mind is: Being a search engine, doesn’t Elasticsearch already offer caching? What’s the need of an application layer cache?

Elasticsearch offers no native caching capabilities for search queries. For filters and aggregations, it’s a mixed bag — with limited support and configurability. Let’s walk through Elasticsearch’s caching mechanisms to get a better sense of this.

Elasticsearch’s Caching Mechanisms

Known for being one of the fastest search engines, Elasticsearch offers four ways to cache data. We will look at each of them and how they compare next.

1. Filesystem Cache

The Filesystem cache (aka the OS cache) technically resides outside of Elasticsearch, but for completeness and accuracy, we will start from here.

Elasticsearch recommends only reserving up to half of the machine’s memory for its heap usage. The other half is reserved for the filesystem cache. This cache only memory maps Lucene term dictionary, norms and doc values — these can thought of as intermediate states in retrieving results but not the results themselves. Also, this cache offers almost no configurability on the user’s end. You can read more about it over here.

2. Shard-level Request Cache

The Shard-level request cache stores results, but is local to a shard and only caches when Search and Filter Query results aren’t involved (i.e. size is set to 0). Typically, this means caching Aggregation only queries. It offers a good level of configuration for this use-case. You can read more about it over here.

3. Query-level Request Cache

The Query-level request cache operates at a node (aka machine) level and stores results of queries used in the filter context. This cache leaves out search queries (i.e. any query where relevance is critical) as well as aggregations. It offers limited configurability: you can enable/disable this cache per index as well as configure max heap space it utilizes. You can read more about it over here. So, how does this particular cache measure up?

4. Field-data Cache

The Field-data cache is primarily used when sorting on or computing aggregations on certain field types. This cache resides within the heap and its size can be configured but not much else. Although, its use-case coincides with shard-level cache, it is global in nature and only caches these data-structures (think of intermediate states) vs results themselves. You can read about its configuration over here.

In conclusion, none of the native Elasticsearch caching mechanisms serve the search queries use-case and there are limitations to the configuration mechanism.

If you don’t want to take my word on the above ;), you can also read Elastic’s recent deep dive on how caching works.

Appbase.io’s Application Layer Cache

Appbase.io’s cache meets the missing gap of serving results from a configurable cache across search queries, filter queries and aggregations.

This cache operates in a complementary manner to Elasticsearch’s own caches — it caches direct results and doesn’t change any of the existing endpoint behaviors. It operates on the ReactiveSearch API only— a declarative API to querying Elasticsearch — used by appbase.io customers, and ReactiveSearch & Searchbox client libraries (which collectively see over 500K downloads/year).

It also comes with a convenient point and click UI to manage the cache preferences. Read more about it over here.

Benchmarking: 3..2..1 GO!

Below is a benchmark showing the search performance with cache enabled and without cache enabled. For this benchmark, we performed 10K search requests over a period of 5 minutes. Here, we assume 50 unique search terms, i.e. most search queries are different users searching for some of the most common thing (to simulate an e-commerce search scenario).

With appbase.io cache, the end-to-end network latency drops to an average of 47ms (a 120ms improvement). While the absolute numbers depend on factors such as geographic distance between the client who’s requesting for data and the server, their connection strength, the comparison between the two modes removes these variables out. The key takeaway here is that users can expect to see their latencies drop by the same time that Elasticsearch itself takes to compute the search queries.

(Not) Yet Another Cache

Alright, alright, Elasticsearch’s native caching mechanisms may be limited. However, can’t I just use a 3rd party caching solution instead?

One could go for this route: however, then they would lose two important configuration parameters.

1. Index selection — You typically only want to enable caching for indexes that are serving application data.
2. Per request granularity — You typically want to turn caching on / off for certain requests (e.g. making an expensive low-frequency query, when getting live data is more important than fast data, when profiling queries)
3. Control Plane UI — You also typically want a GUI to manage your cache configurations.

Sure, these are things one can implement on their own on top of a 3rd party caching solution. In practice, however, the time to implement and maintain the additional business logic turns out to be impractical. To the extent that our customers with the most demanding app search use-cases have refrained from going this route.