“With databases, the conversation always started with ‘what are we able to do?’. I rarely find myself asking what Rama is able to support, and rather ‘how?’. The requirements of the application dictate how we utilise the platform, not the other way around. Rama as a tool allows us to think product first, while still delivering highly optimised and scalable features for specific use cases, something that would not have been possible without a much larger team.”
William Robinson, engineer at Multiply
Multiply rebuilt their entire backend from scratch in the past year using Rama. Their prior implementation was built with XTDB, and their prototype before that used Datomic.
Multiply is an AI-powered platform for collaboration and co-creation. Expert AI agents plan, communicate with other agents, leverage tools, analyze files, conduct online research, and report back when the work is done. Here’s a short video introducing the platform.
A few aspects of the application are demanding of the backend. The application is highly interactive, so low-latency reactivity on pretty much anything that changes on the backend is critical. Users create complex workflows that execute asynchronously, with the state of these workflows needing to be managed in a fault-tolerant way. There’s also many features which are fully synchronous with user interaction, with a wide variety of indexing and query requirements.
Datomic was appealing for the first version for its immutable data model and expressive Datalog-based query language, attractive and intuitive concepts for most Clojure programmers. They soon switched to XTDB, as even though XTDB is similar to Datomic, its more flexible ways of representing data helped reduce their codebase size.
The problems that led Multiply to switch to Rama were the same ones typically faced with databases. Almost every database, not just Datomic and XTDB but also Postgres, MongoDB, Redis, Cassandra, etc., supports a small set of data models. Most databases support just one. A data model dictates how a database indexes data and the database API is oriented around what’s possible with that indexing strategy.
The first problem Multiply had was understandability. Squeezing everything into a fixed data model was unnatural, and they had to build lots of “support infrastructure” to deal with the impedance mismatches:
“I came to the realisation that we didn’t understand the system we’d built. We understood each part in isolation, but we didn’t have the tools to “see” the system (through the lens of our domain model and business logic) operating.”
Henrik Eneroth, CPO of Multiply
“It often felt like we were working against the database, and that our efforts went towards designing our product around it.”
William Robinson
The lack of understandability made everything take longer, whether fixing bugs or developing new features. This severely impacted their iteration speed, which is particularly harmful for a startup.
The second problem was the difficulty of achieving acceptable performance and fault-tolerance, also common problems when using a database that can only index a limited number of ways:
“We ran into bottlenecks for running deep live queries. While we found acceptable levels of latency on write, we were unable to get fault tolerance across multiple nodes. We spent many long months on trying to get it to work. In the end, it was a pricey, tiring and failed effort.”
William Robinson
Multiply understood the potential of Rama to fundamentally solve these problems as soon as it was announced. Rama being a platform for both storage and computation, with an indexing model based on data structures that supports infinite data models, could let them mold their infrastructure to fit their product rather than continue to contort their product to fit their infrastructure:
“Rama seemed like it would remove a lot of the aforementioned support infrastructure that we’d built around XTDB. This was a huge bet of course, but it paid off on removing a ton of the custom plumbing we needed.”
Henrik Eneroth
Indexed datastores in Rama are called PStates (“partitioned state”). Multiply created many PStates to support their application with a wide variety of shapes, many of which are very different than how a database could represent data. PStates are distributed, durable, and replicated, making them suitable for any use case that a database could be used.
Let’s take a look at some of the PStates powering Multiply. The following definitions are used in the PState schemas:
1 | (def Sub String) |
One of their simplest PStates tracks the account ID for each login subject (e.g. “google-oauth2/12345” is a login subject using google-oauth2 with user ID 12345 for authentication):
1 | (declare-pstate account-mb $$subs {Sub AccountID}) |
This PState is called
$$subs
(PState names always begin with
$$
) and is equivalent to a distributed key/value database. Its schema is a map from
Sub
to
AccountId
.
Another PState tracks information for every registered email:
1 | (declare-pstate |
This is a “map of maps” schema, where the inner maps have a fixed set of keys each with their own schema. This is equivalent to a distributed document database.
Then there are many PStates which index data in ways that no database can. These PStates are finely tuned to exactly what’s optimal for Multiply’s use cases. For example, this PState stores LLM responses as tokens are received:
1 | (declare-pstate |
Every organization can have an arbitrary number of LLM requests, and each LLM request can receive an arbitrary number of tokens. The inner
map-schema
tracks all the LLM requests for an organization, and the inner
vector-schema
is the list of response tokens for that LLM request. This PState uses the
:subindex?
flag, which tells Rama that value should have its elements indexed individually instead of the whole value being serialized as a whole. This lets those nested data structures be of huge size while still being able to read and write to them quickly.
This PState can handle many queries in less than one millisecond: get the number of LLM requests for an org, get a range of LLM requests for an org, get the number of tokens streamed in for a particular LLM request, get a range of tokens for a particular LLM request, and many more. Because Rama has a generic way for doing queries of any complexity on PStates of any structure (paths), all these queries are tiny amounts of code to implement.
This PState is highly specific to Multiply’s use cases. Multiply was able to structure all their PStates to match their domain model exactly, including making use of the freedom that Rama gives to use any types they want. Values within PStates can be primitive types, Clojure defrecord’s, or anything else. They don’t need any adapters for their datastore to manage the mapping from their domain model to the datastore, as is typical with databases (e.g. ORMs).
Finally, since Rama applications are based on event sourcing, with all data coming in on distributed logs called “depots”, Multiply always has the flexibility to recompute PStates from scratch if they need to aggregate or index it in new ways in the future.
Multiply had this to say about the newfound power Rama gave them:
"Rama put us back in control, by writing code that can express what would previously require us to deploy an entirely new service that we probably would not fully understand. Its abstractions allow us to tap into the power of a small team, by expressing architecture through code, and to focus on business logic instead.
It’s trivial to write custom PStates to support certain features or views, contrary to having convoluted queries or transformations. ‘This could just be a PState’ is something we exclaim quite a bit. It’s also nice to not have to clutter existing PStates."
William Robinson
All the intricacies of their application could be expressed naturally with Rama’s dataflow API for reacting to events as they come in. Whereas before business logic was spread across a web server and a custom background worker, with Rama they were able to express all their logic – both synchronous and asynchronous – in a unified system. Besides how much this simplified thinking about the system, it eliminated all the work to orchestrate the deployment of multiple systems for storage and computation.
Rama also enabled them to build things that were not possible with XTDB and would have been huge efforts without Rama:
“The LLM task runner would not have been possible with XTDB. Without Rama, we’d need more than one database. There’s a great deal of work we haven’t had to do for the capabilities that we now have with Rama. It’s easy to have the cake and eat it too with Rama.”
Henrik Eneroth
Multiply’s implementation is a single module with:
- 17 depots
- 19 PStates
- 7 microbatch topologies
- 6 stream topologies
- 42 query topologies
They run their application on five nodes with a replication factor of two, deployed to Fly.io performance-4x machines.
The depots separate incoming events according to the entities they affect. For example, one depot is for changes to accounts, another depot is for changes to organizations, another depot is for changes to collaborative spaces, and so on.
Their frontend uses Rama proxies heavily for reactive queries, which is the basis of how they enable their application to be so interactive.
Their Rama deployment required minimal DevOps work:
The DevOps and backup/restore experience with Rama is truly next level. Compared to my experience in running other data systems, the complexity of setting up and running Rama in production is much lower. And due to the power and flexibility of the programming model plus the tremendous horizontal scalability, you probably don’t need to run any other database, which simplifies things even more.
Oskar Boëthius Lissheim, CTO of Multiply
Deployment is handled completely by Rama’s built-in and fault-tolerant facilities to launch, update, and scale modules. Rama has comprehensive monitoring built-in, which Multiply augmented with basic node telemetry to track things like disk space and CPU load. Rama’s integrated backups feature made it trivial to keep all state continuously backed up to S3.
Rama multiplied Multiply’s capabilities and productivity. Rama attracted Multiply not for its scalability, but for its ability to fundamentally reduce complexity. That their application will scale to any read/write load in the future is a big bonus.
You can get in touch with us at consult@redplanetlabs.com to schedule a free consultation to talk about your application and/or pair program on it. Rama is free for production clusters for up to two nodes and can be downloaded at this page.