Transforming Identity and Access Management with Event Sourcing

In the digital realm, managing who gets access to what is a huge deal, and the foundation of a robust end-to-end identity infrastructure becomes a crucial component for your project. Identity and access management (IAM) systems, the cornerstone of this infrastructure, act as the digital gatekeepers, meticulously managing who gets access to what.

They’re not just about guarding the entrance; they’re about ensuring the right people have the right access at the right times. But there’s an aspect where many IAM systems are missing a trick: keeping a detailed record of user activity once they’re in. Imagine a bouncer who not only knows who came in but also remembers every move they made inside. That’s where event sourcing comes into play, and it is a game-changer in the IAM space.

While essential, many IAM systems have yet to harness the full potential of event sourcing. This integration is not just an added feature; it’s a crucial upgrade for any organization serious about digital security. With event sourcing, IAM systems gain an unparalleled depth of insight and control. This approach is not only invaluable for troubleshooting but also provides a deeper understanding of user behaviors and system interactions.

Event Sourcing for Integrity and Compliance in IAM

Think of event sourcing as capturing and storing every action a user takes as an event, which is then stored in a sequence in chronological order. Events are not merely passive entries but are treated as first-class database objects, representing the core truth of the system. This approach marks a significant departure from traditional logs, which simply record activities.

Instead, events in event sourcing actively define the state of the system at any given moment. If an event does not happen, the state actually does not change. If an issue arises, you can trace back through these events to pinpoint where things went haywire.

The system can then produce an audit trail that is concerned with how these events (or a subset of them) are presented and accessed for review. It’s the mechanism through which the recorded events are made available for auditing, security analysis and compliance checks. In today’s world, where data breaches are as common as coffee spills, having robust audit trails is more than just a security measure. It provides transparency and accountability, showing stakeholders that your organization doesn’t just talk about data security, it enacts it.

In essence:

• Event sourcing: An architectural pattern to persist data as contextual events to an append-only log instead of the current state.
• Audit trail: Audit logs are detailed records that capture events or transactions happening within a software system. They provide a detailed history of all changes made to the system, including who made the changes, when they were made and what was changed.

In an IAM system, event sourcing would continuously capture all state changes as events related to user identities and access rights.

What Takes Place in an Event-Sourcing IAM System

Take an online retail platform, for example. With event sourcing, every action a customer takes, from registering, stepping through the login process, updating user profile, resetting password, to making authenticated requests such as making a purchase via protected API calls is tracked.

This precise recording helps the business build a comprehensive customer profile, spot trends and tailor the user experience more effectively. Event sourcing ensures data integrity. If a customer disputes a transaction, the business can swiftly consult the event log to verify the details. It also bolsters security. Unusual patterns, like repeated failed login attempts, are flagged, helping to prevent fraud and unauthorized access.

With regulations like GDPR demanding strict data handling, audit trails provide a transparent record of user consent and preferences. Imagine a customer changing their communication preferences; the audit trail captures this change, ensuring the business respects the customer’s wishes and stays on the right side of the law.

How ZITADEL Implements Event Sourcing

As discussed, the integration of event sourcing and audit trails can transform a standard IAM system into a robust, security-focused powerhouse. This is precisely why ZITADEL, an open source identity and access management solution, decided to implement event sourcing right from the start.

A Blend of Event Sourcing and CQRS

ZITADEL’s software architecture stands out for its integration of event sourcing and Command and Query Responsibility Segregation (CQRS). CQRS is all about splitting the two main concerns of any system — the write operations (commands) and the read operations (queries). This division allows each function to operate independently and optimally, without tripping over each other, a pattern that smartly acknowledges that the way you add and update data isn’t necessarily the best way to retrieve it.

The architecture of ZITADEL is strategically designed to be eventually consistent, capitalizing on the strengths of event sourcing and CQRS to maintain an Event Store, which is the authoritative repository for all computed states and is maintained with transaction safety to ensure the orderly execution of operations.

It’s important to note that while the system ensures eventual consistency overall, operations by individual IDs are consistently handled, ensuring reliability and precision. This distinction mainly applies to list operations, which are managed under the eventual consistency model.

The Event Store, following the event sourcing pattern, diligently records every operation in ZITADEL as an event. This allows ZITADEL to construct a chronological ledger of user interactions and system changes, providing an extensive audit trail. Every user registration, authentication attempt, or permission change is captured and preserved ensuring that the state of any resource at any point in time can be reconstructed.

This contrasts with other systems that rely on writing logs, which can be challenging to ensure completeness due to the separation of systems and potential gaps between what is stored in the IAM system and what is logged. Furthermore, users can seamlessly query the audit trail through ZITADEL’s versatile Event API.

Complementing event sourcing, the CQRS pattern separates ZITADEL’s read and write operations, enhancing the system’s scalability and performance. The write side, or the “Command” part of CQRS, handles all state-changing operations in the system, passing each command through rigorous validation to ensure transaction safety and system consistency.

On the flip side, the “Query” aspect deals with all data retrieval requests, optimized for speed and reliability, even during spikes in load.

An Overview of ZITADEL’s Software Architecture

To fully grasp the mechanics, it’s crucial to understand a few key terms:

Events: Represent the occurrence of actions in the system that get permanently recorded. Events form the basis for the system’s current state. The current state of data is the aggregation of all events. This cumulative process ensures that the summary always reflects the latest state, Here are some possible events:

• user.added
• user.changed
• project.added
• user.password.checked

Aggregates: Collections of related events that together determine the state of an object, such as a User, Organization, Project etc. They serve as transactional boundaries.

Projections: Computed objects derived from the event log. These objects are optimized for read operations and queries. Instead of reconstructing the state from the event log each time, projections provide a more efficient way to access current state information for various entities like list of users, projects, auth requests, etc., via searches.

Frontend Layer

ZITADEL encapsulates all essential components within each binary, simplifying operation. This all-in-one approach includes serving the API, rendering the GUI, and handling the background processing of events and tasks (abstraction layers written in Go, which encapsulates all the functionality of the Event Store away from the business logic). The business logic only interacts with events through pushing and filtering capabilities.

Services Layer

At the services layer, ZITADEL exposes all components potentially interfacing with consumers, including an HTTP server for managing the ZITADEL Console and serving static assets, alongside API servers, including OpenID Connect, OAuth, SAML, Authentication, Management, Administration, System, and Asset APIs, that support various protocols like gRPC and REST.

Core Layer

The command handler processes all operations that alter a resource, generating the necessary commands for validation. This stage ensures that each action, such as a user name change, is legitimate before creating events that reflect the changes. These events are then processed either in real time by an in-memory pub-sub system or through a spooler for background processing. The pub-sub system keeps query views up-to-date, and spoolers ensure no significant lag occurs behind the Event Store.

Projections within ZITADEL normalize data for queries or analysis, invoked by either the spooler or the pub-sub system. They generate normalized objects that are stored within the query view and its storage layer. The query handler then addresses all read requests, delivering rapid response times and high availability.

Storage Layer

The storage layer, crucial for a stateless system like ZITADEL, is responsible for data distribution across servers, data centers or regions, ensuring strong consistency for commands and optimal query performance.The event storage mechanisms depend on the type of database used, especially when dealing with different databases because they do not handle commit timestamps in the same way. This has implications for the ordering of events, a crucial aspect of event sourcing. ZITADEL favors PostgreSQL for its alignment with the platform’s needs, offering data residency options and disaster recovery capabilities.

Navigating the Audit Trail

Users can access event data through the ZITADEL Console or Event API, providing a complete view of all stored events. This allows for customization and filtering based on the user’s needs. The event data is beneficial for various personas, such as end users, who can track their login activities, or administrators, who can monitor user activity to make informed decisions, such as scaling resources or investigating security incidents.

The Console View

Last Changes View

Within the Console, users can review the most recent changes to objects, such as user profiles, organizations or projects, providing immediate insights into recent modifications. In the following screenshot, you can see an example of last changes on a user object. The same view is available on other objects such as organization or project.

Events View for Administrators

Administrators have the luxury of filtering and reviewing all events across an instance directly within the Console, ensuring comprehensive oversight.

The Event API

For a programmatic approach, ZITADEL’s Event API allows full access to all events and audit data, enabling integration with custom business logic. The cURL request below shows one way of invoking the Event API:

The API exposes various types of event data, including aggregate types (such as user, organization) and event types, which describe specific operations that have occurred. For example, with the search endpoint, admin users can query and retrieve a list of events based on specific filters such as sequence number, event types, aggregate IDs and more. This detailed event data provides insights into the operations and state changes within ZITADEL.

Moreover, users can use the Event API to funnel data into external systems. Additionally, ZITADEL provides the flexibility to write events to stdout for processing as logs or even send events directly to external tools via HTTP requests, though the latter is not recommended due to potential back-pressure and increased processing times.

Key Benefits and Uses of the ZITADEL Event Store

• Historical insights: For audit or analysis, selecting events within a specific time range can provide detailed insights into past changes. ZITADEL allows users to view resources at any given point in history, thanks to its unique data storage methodology. One can review past data within the context of the system’s entire state.
• Future projections: The ability to replay events since installation facilitates easy computation of future projections with new business logic. To bolster user account security, ZITADEL users can leverage its event store to train predictive models. These models are designed to proactively identify and prevent potential attacks, enhancing overall account protection.

Ongoing Enhancements

• Metrics and standard reports: ZITADEL has plans to provide streamlined reporting for better insights. There is a need for dashboards and reports to track user activity, app requests, compliance, login violations, threats and audit trail queries. The list also includes but is not limited to access logs, execution logs and underlying components like CPU/memory usage. Queries can include failed logins per client/project, top users with authentication failures, most active Identity Providers, last execution logs and more.
• New APIs: A metrics API will be provided for easy access to dashboards and reports, and an audit API for filtering multiple resources at a time. The functions include streaming events into a metrics model, allowing for time dimensions and metrics like user registration, successful logins and failed logins. A scrapable endpoint for binary runtime metrics, a Prometheus-compatible API for time series and a LogQL-compatible API are also included in the roadmap.
• Feedback loop and threat detection: ZITADEL will provide clients’ end users access to their event log, enabling them to identify and report any unusual activities. The upcoming features are designed not just for manual reporting but also to train advanced AI models. These models will actively monitor and flag anomalous behavior, including the ability to alert on historical events. Key features will include users flagging suspicious events, notifying users or managers of sensitive actions, generating comprehensive systemwide audit reports, creating rule-based audit events and implementing stricter authentication measures when unusual activity is detected. This AI-driven approach aims to proactively identify potential threats, ensuring a more robust and responsive security framework within ZITADEL.

Summary

Event sourcing in IAM is a game-changer for businesses because it offers a detailed and unalterable history of user behaviors and transactions, which is vital for understanding and managing identities. With event sourcing, along with CQRS, IAM solutions can allow for the maintenance of detailed audit trails far surpassing traditional systems. A built-in audit trail facilitated by event sourcing is more than just a feature; it’s a commitment to transparency and integrity. By meticulously appending every event and change, an IAM system ensures that users can trust their data’s accuracy, delve into their historical records with confidence, and leverage comprehensive insights for future developments.

About ZITADEL

ZITADEL is an open source identity and access management (IAM) solution designed to enhance security for applications and services. It supports various authorization strategies, including role-based access control (RBAC) and delegated access, making it a great choice for both business-to-consumer and business-to-business scenarios. ZITADEL offers a cloud-based SaaS option and can also be downloaded for self-hosting, offering flexibility. Its primary goals are to provide seamless authentication and authorization, facilitate auditing, enable custom extensions, adhere to standards like OIDC/OAuth/SAML/LDAP, and ensure ease of operation and scalability. The community and team actively contribute to its development and support, making it a powerful tool in the realm of identity management.

Further Reading

• What is Event Sourcing?