Event Sourcing

What is Event Sourcing?

In a traditional way of storing an application’s state, we capture the current state and store it in some relational or NoSQL database. While this approach is really straightforward, it does not provide a way of deducting how we got to that current state. Of course, one may argue that we can have a separate model for keeping the history of actions that lead to the current state. Still, besides the additional complexity, these two models could easily go different paths and start being inconsistent (which is basically an update anomaly).

Event Sourcing is a way of storing an application’s state through the history of events that have happened in the past. The current state is reconstructed based on the full history of events, where each event represents a change or fact in our application. Events give us a single source of truth about what happened in our application. It is especially beneficial in applications that need to provide a full audit log to external reviewers. The current state is called Materialized state in some resources. The following figure represents the reconstruction of materialized state based on the history of events.

 

Example of Event Sourcing

Let's see an example of how Event Sourcing differs from Traditional Storage. In a Traditional Storage system, we only know that we have ordered a pizza and a cola. In Event Sourcing, we see that a user selected a pizza, selected a cola, selected an ice cream, and deselected an ice cream. Information about the selection/deselection of ice cream is not present in Traditional Storage. With Event Sourcing, we can reason why a user deselected an ice cream, was the price too high, or some other reason. The point is that we didn't lose that information, and we can benefit from it in various ways. Later on, we can see that a user confirmed the order.

Online Order	Traditional Storage	Event Sourcing
Select Pizza	Order 123; Selected Pizza	T1: Selected Pizza
Select cola	Order 123; Selected pizza, cola	T1: Selected Pizza  T2: Selected cola
Select ice cream	Order 123; Selected pizza, cola, ice cream	T1: Selected Pizza  T2: Selected cola  T3: Selected ice cream
Deselect ice cream	Order 123; Selected pizza, cola	T1: Selected Pizza  T2: Selected cola  T3: Selected ice cream  T4: Deselected ice cream
Confirm order	Order 123; Ordered pizza, cola	T1: Selected Pizza  T2: Selected cola  T3: Selected ice cream  T4: Deselected ice cream  T5: Confirmed order

 

Axon makes things easy when it comes to implementing and scaling out event-sourced Java applications. Axon Framework provides a programming model and necessary abstractions for working with Event Sourcing. On the other hand, Axon Server abstracts the distribution of messages (command, queries, and events) and storage of events.

 

Technical Insights

Storage

The Event Store is responsible for storing events. Since events are not modified (an event is a fact that something happened and facts cannot be modified), an Event Store should be optimized for appends. Event ordering plays a vital role in event-sourced systems - as many times as we are reconstructing our materialized state, we want to arrive at the same result. Axon Server is the default choice in Axon and the best way to start. It is highly optimized for storing/retrieving events. If you really want, alternatively, you might consider an RDBMS or a NoSQL database. Axon has out-of-the-box implementations that support JPA, JDBC, and Mongo.

 

Event Replaying

In order to have materialized state reconstructed, we need to trigger an event replay. That means that all events will be handed over to event handlers which will process them and build the current state of an application* to apply our business logic. This gives us a powerful mechanism to diagnose (debug) any problems that happened in the past, replay events up to a point when the incident occurred, and assess the current state. If we encounter a problem with application logic, we could fix it and replay events.

We should be cautious about event replays. Think of the case in which the event handler contacts some external system (sends a bill to be paid). Certainly, we don’t want to replay this event. Gateways to other systems can be disabled in case of event replays to avoid this kind of problem. Axon Framework provides more granular control to event handlers in terms of which events get replayed and which don’t. Also, it is possible to monitor the progress of the replay procedure.

 

Event Sourcing and CQRS

Event Sourcing is a natural fit with CQRS. Typically, the command model in a CQRS based architecture is not stored, other than by its sequence of events. The query model is continuously updated to contain a certain representation of the current state based on these same events.

Instead of reconstructing the entire command model state, which would be a lengthy process, we separate the model into aggregates, parts of the model that need to be strongly consistent. This separation in aggregates makes models easier to reason about, more adaptable to change, and more importantly, it makes applications more scalable.

Reloading an aggregate state reliably involves re-applying all past events to an aggregate. After appending a large number of events, you can imagine that this can become a lengthy process. To reduce loading time, you can take a snapshot after a certain amount of time, several events, or some other criteria. A snapshot in this context represents a current state of an aggregate. Next time we want to do a replay, we’d start from a snapshot and replay only the events that came after taking the snapshot. Snapshotting is usually an asynchronous process, so it does not interfere with regular event processing. In Axon, building snapshots is a matter of configuring when they should be taken. The following figure shows how snapshots relate to the events in the Event Store.

 

Upcasting

It is fair to assume that event schema is going to change in the future. Event handlers usually can cope with these changes, but we pollute business logic unnecessarily. For such purposes, we introduce upcasters. Upcaster is a function that takes an event described using an old schema and transforms (upcasts) to the event described using a new schema (there are certain cases where you want to combine several events into a single event or to spawn several events from a single event). It is not uncommon to have several upcasters for a single event type; we chain them into an upcaster chain used during event replaying. The event replay mechanism takes events, runs them through corresponding upcaster chains, and hands them over to the event handlers.

When upgrading event schema, certain practices make the transition run smoothly.

• Don’t remove fields; deprecate them.
• Provide default value when introducing a new field
• Don’t change event type; introduce a new event

When applying an Event-Driven Architecture on a large scale, we recommend having a deprecation policy in place, which describes how teams are expected to deal with the depreciation of elements and under what circumstances these elements may be removed.

 

Security Concerns

Events are immutable (not to be updated), and all information stored in these events must be available all the time. With some regulations (like GDPR, for example), a user can require the software system to “forget” one’s data. This request contradicts the nature of Event Sourcing. Fortunately, there is a technique called cryptographic erasure, which uses cryptographic algorithms to encrypt sensitive data. Keys used for encrypting data are stored alongside the event store. So, when we want to store sensitive data into an event, we encrypt it with the given keys. Each time we do an event replay, data is decrypted and handed over to the event handlers. Once a user requests data deletion, keys are deleted, and sensitive data cannot be accessed ever again. Axon Server provides an enterprise pack to deal with this kind of challenge.

 

Benefits

The benefits of Event Sourcing are a bit scattered around this article, so it wouldn’t hurt to sum them up in this chapter.

The full history of interactions with our application is stored in the Event Store. We could apply various machine learning algorithms to extract information from these interactions that matter to our business.

The agile approach to building software systems requires that we should be able to adapt to any change coming along the way. The ability to replay the event stream from the beginning of time with new business logic means that we don’t have to worry about decisions we make (apart from which events are important to be stored). We can always fix the behavior later. Introducing new views to our event stream means adding a new component with event listeners to the solution.

To comply with certain regulations, a software system must be required to provide a full audit log. Event-sourced systems give us exactly that, a full audit log, and we don’t have to provide any additional information to the reviewer. One additional report is built up from the event stream, and we’re ready to go.

We all know how difficult it is to investigate an incident that happened in production. It requires a lot of logs digging and reasoning about the state that the system was in then. Event sourcing gives us a way to replay events to a certain point in time and debug the application in a state where the incident occurred. We don’t have to worry about whether we put the correct log level or logged all necessary paths to figure out the incident.

If we figure out that the problem is in our business logic, fixing the problem may be difficult. The event history that is already there is not meant to be altered, so if we want to change that, we might want to use upcasters to change the event structure to fix the incident.

 

Conclusion

In software systems where our business can benefit from the history of events, Event Sourcing comes as a natural solution. One of the key benefits of Event Sourcing is the evolvability of software systems. When we discover a new reporting component that needs to be added, we can write it up, replay the historic events, and run it. This is highly beneficial in blue-green deployments when zero downtime is enforced.

Integration with external systems can be done using events. In such scenarios, events are our API, and the external system must understand them. Of course, we might not publish all events to the integration event hub, only publicly important ones.

* If we are applying CQRS (Command Query Responsibility Separation) practices, we could rebuild our command model and query model as well

Axon’s product suite provides an easy way to start up and scale event-sourced Java applications.