It’s Time to Move On From CRUD. There is a Better Way.

Over 20 years ago, Starbucks coffee shop in Canberra City regularly played host to lunch break discussions dissecting the state of the art in software development. I have fond memories of discussing and debating with my colleagues on architectural patterns, development tooling and more esoteric subjects like visualisation of code and executable specifications.

At the time, the CRUD (Create-Read-Update-Delete) pattern was heavily used in applications. It generally came with lots of data grids, lists and edit forms, and often an almost 1–1 correspondence between form content and relational data tables.

Distributed applications had also become the norm. Single user applications with a local database had all but disappeared in favour of the client / server paradigm. The result was a massive increase in the complexity involved in building applications. Such a significant gap exists between the skills required for client and server programming that entirely new programming roles were born — front end, back end and “full stack” developers.

Aside from a very poor user experience, these systems did not scale with complexity very well. As features were added, the “business logic” entities became overloaded with responsibilities, meaning and size. Enterprise systems with dozens of 1,000+ line classes are no fun to maintain!

While user experiences are generally a tad improved today and the skill gap between client and server programming is not as wide, CRUD and related patterns remain widely used¹². Software development has not yet enjoyed the same sort of advancements that the cloud brought to infrastructure management.

The Rise of Event Based Systems

While also having existed for decades, event based architectures have surged in popularity in recent years³⁴. Patterns such as command query responsibility separation (CQRS) and event sourcing allow highly decoupled code structures that scale well in terms of both complexity and performance.

The “Kappa Approach”⁵ intelligently routes events generated in the user interface through reducers. A reducer is a function that accepts a current state value along with the next value in a stream of values and returns a new state value. This pattern can be used to transform a stream of events into view models that can be consumed directly by user interfaces.

A notable observation about this style of architecture is that complete object models become directly derived from events. In fact, when you apply similar lines of thinking even to more traditional architectures, it is an inevitable conclusion that all object models are ultimately derived from events.

Let me give you some examples.

• Your bank account is a series of events that started with creation and has a balance that is the sum of all transactions against the account.
• Social media boils down to a bunch of events — posts, comments, likes, shares, etc.
• Utilities management companies (gas, electricity, etc.) manage events — the installation and maintenance of infrastructure, connection of premises, meter readings, payments, etc.
• Customer relationship management systems track the actions of company representatives and customers.
• Every sport and game consists of a series of significant game events that culminates in a winner being determined from those events — often a “scoreline”.

When you start applying this thinking to the visible object models in systems around you, you start to see how almost every system can be thought of as entirely derived from input events. Almost everything we can model with software is a product of interactions.

It turns out that making event messages the primitive building blocks of a system, as opposed to data rows or documents, opens the doors to some very cool abstractions.

Building the Foundations

We are essentially saying that there is a set of steps for transforming a series of input event messages into the data outputs for a system. Taking this further, there exists a formal language for expressing those transformations.

We can use libraries such as ReactiveX or @x/expressions to explore what such a language might look like. These libraries provide a low level language for transforming streams of events using a set of operators.

For example, an expression for the average rating given to products on an online shop might look something like:

productRating: o => o.topic("product reviewed").average("rating")

These expressions are essentially the reducers mentioned earlier. The two parts are called operators and are individual reducers that are chained together.

Similarly, the current stock level of a product might be expressed as:

stockLevel: o => o.compose(
  o => o.topic("stock received").sum("quantity"),
  o => o.topic("item shipped").sum("quantity"),
  (received, shipped) => received - shipped
)

These pieces of vocabulary can be composed into higher level structures such as view models for binding to user interface elements, constraints that can apply contextual validation to messages, security rules, etc. The provided operators are comprehensive enough to produce output of any desired structure.

productView: o => o.assign({
  "...details": o => o.topic("product").accumulate("details"),
  "rating": o => o.productRating(),
  "available": o => o.stockLevel()
})

As expressions are derived directly from events, they can be updated in real time and seamlessly synchronised across consumers in an eventually consistent fashion.

Processes — The Bigger Picture

A shift to making events the atomic building blocks of a system also opens the door to better ways of modelling the state of system artefacts over time.

Statecharts provide a formal mechanism for modelling state transitions from events. They are essentially reducer functions and work well with the architecture described above.

For example, a typical online shopping experience consists roughly of products being added to a basket, the order submitted, invoice paid, order picked / packed, shipped, etc. Reaching each of these states might trigger other processes or call external systems.

Statecharts can be used to model the longer lived workflows in a system, potentially involving many actors, as well as short lived processes like application navigation.

They can be visually designed and potentially serve as a powerful communication and validation tool.

Providing Services

Naturally, there are things that fall outside the scope of this approach. For example, complex algorithms and CPU bound operations like numerical analysis, 3D rendering, text processing, etc.

These can be written in the host language and “shelled out” to, exposed using a plugin architecture, or otherwise integrated with.

A Simply Better Abstraction

The combination of expressions describing a system’s data model and statecharts governing a system’s state transitions provides a rich abstraction capable of expressing the vast majority of business and consumer systems.

The abstraction ends up being significantly simpler than more traditional approaches. The separation of client and server is gone — data persistence, distribution, synchronisation, transport / serialisation, all seamlessly handled. Consumers can be updated in real time without an additional line of code.

Everything is expressed using consistent language — views, state machines, constraints, security rules, etc., all leveraging the same vocabulary. Less code and far fewer technologies to understand.

Models are loosely coupled projections, making them easy to change and extend. New features can be added and old ones updated, the new models automatically synchronised to the current point in time.

The declarative nature of expressions and statecharts enables all sorts of optimisations. Read models can be automatically cached for high performance user interfaces. It is possible to dynamically partition and scale out to multiple servers seamlessly.

The Next Step — Natural Language

It would seem the ideal scenario that systems could be constructed from plain language. The development process would essentially be summarising key aspects of the problem domain in plain English. It has been a goal of the industry for decades to construct software from requirements specified using diagrams and a formal language.

DeepMind’s AlphaCode recently proved that machine learning is capable of producing code from problems specified in plain English. The proprietary technology demonstrated producing low level code for individual algorithms described (somewhat obscurely) in a coding competition format.

Using a higher level language as the output of machine learning algorithms paves the way for enterprise scale systems to be largely generated from plain language.

Importantly, the abstractions outlined above provide constructs that correlate with both concepts of “verbs” and “nouns”, instead of the mostly noun-driven modelling inherent to CRUD. Given sufficient training data, they could potentially be generated by a general purpose language processor like OpenAI’s GPT-3.

Ultimately, the developer’s role will transition to being more akin to an analyst — gathering requirements, resolving ambiguity or conflicting information, and concisely summarising to produce an “executable specification”. This specification bridges the gaps in the ability of machine learning systems to understand the world, and communicates how the system works in a way all stakeholders can understand.

The State of the Art

The expression language used above represents a step forward over current technologies in terms of composability and portability. It is designed to model entire event domains declaratively without the need for imperative code.

Expressions are fully serialisable, allowing them to be persisted to a data store or transmitted over a network. @x/socket is a complementary library that facilitates seamless and reliable distribution of expressions and other observable types.

The xstate library is a powerful and mature statechart implementation, complete with a visualisation tool, seen above.

Unify is an experimental new framework that combines all of these technologies. It is an opinionated framework that seamlessly and securely distributes expressions and statecharts over any socket. There are many tutorials and samples available, including a real time conference feedback tool.

The industry is on the cusp of a productivity revolution thanks to a shift towards event based thinking. Improved tools and frameworks are making the entire paradigm accessible to all developers — tools and frameworks that should now be part of every developer’s tool kit.

Are there tools or patterns you use to build event based systems? Let us know in the comments below.

Footnotes

[1]: For example, IBM’s RedHat released a Node.js reference architecture that includes a GraphQL CRUD system.

[2]: There is plenty of searchable reading on CRUD, REST and anti-patterns.

[3]: The Javascript implementation of the ReactiveX library, RxJS, boasts almost 40 million weekly downloads from the npm package repository as of April ’22, up from around 24 million the previous year.

[4]: Apache Kafka is a popular open source event processing and distribution platform written in Java. Downloads of the Javascript client from the npm package repository have more than doubled in the last year, from around 155,000 weekly downloads in April ’21 to over 355,000 in April ’22. It is available as a managed service on certain cloud providers.

[5]: Aris Kyriakos Koliopoulos gives an excellent presentation on the Kappa Approach at Flink Forward Berlin 2017.