May 17, 2016 at 00:53Lazy event sourcing or how to live for today
Translation of an article published at Eventsourcing Publications . This article describes some of the ideas used in the Eventsourcing project .
If you read an article by Fowler or similar sources on the subject of event sourcing, you might have something like this in your brain:
The general idea of this approach is that the user (or any other external system) generates commands, we process them by adding the received events in the event store and updating the "state of the world" in the database, the data from which the user requests.
This approach looks simple and beautiful. We have enough data to “replay” events, we have where to request data on the state of the world from and we can use time-tested databases. On the other hand, I noticed that I wanted a little different from the concept of event sourcing. I wanted to avoid foreseeing the future and this model was somehow not very suitable, because I had to write the updated state to my read-only database.
The very idea that we collect intentions for actions and all the related data leads to the idea that you can always look into the past and use information that was not very useful during the collection of this information (but now there are new observations, new functionality, etc.) What does this mean? Do I have to update my database schema to support new structures? Do I have to replay all events to reload the database?
To be honest, I would like to avoid such difficulties.
During my career I have developed enough projects to understand a few simple truths. One of these truths - the most constant thing - is change. In fact, there is no normal way to find out what your application and data model will look like in a few months. Maybe after the rewriting it was all three times from scratch, and even that is in question.
What if I can postpone guessing the future until I will actually be in this very future? Since I record every event, I can always rebuild the “state of the world”, playing all the old events again, but, as I noted earlier, this is a very expensive pleasure.
But what if I can just send requests to search for events that fall under the necessary criteria, right at the time of the request from the user? For example, suppose we have users and they have email addresses. Suppose we created UserCreated and EmailChanged events. What if, instead of replaying all the events for all users (or even for one!), We just look for UserCreated (id:) and the last EmailChanged (userId:)?
Hm. We can request the necessary information about users only when necessary, without replaying events!
If the algorithm for finding out the user's email has ever changed, all we need to do is change the User # email () method. Moreover, until I write the User # email () method itself, I don’t need to know that this is exactly how I will structure the data for the presentation. I just have to write down all the data in my events, if possible without losing anything.
More interestingly, this approach leads to the following conclusion: in fact, there is no universal version of the “state of the world”. Just as we humans perceive the world and the reality around us differently, depending on the context, programs can perceive reality through different prisms and I see no reason why they should be limited in this.
Are there any problems with this approach? Of course. For example, if any of your search queries is time consuming. But, when such bottlenecks are detected, they can be optimized through some kind of preprocessing or caching. It would be nice of course to avoid costly requests, but we live in the real world.
Does this mean that we should not have event handlers that execute during event recording? I do not think this is a good way to respond to changes, especially in domains that are implemented by third-party components. I just don’t see the point of doing this for each event.
Here, by the way, is another interesting consequence of using this approach. With the growing popularity of GraphQL as an API, more and more applications are optimizing the amount of data transferred and the number of queries, requesting and receiving only the minimum that they need. Together with the "lazy" collection of data from events, we never build the "state of the world" on those parts of the data that are not requested.
The above approach is used in the Eventsourcing project , including its extension called Domain Protocol. The project also has an adapter for GraphQL . The project is fully open and licensed under Mozilla Public License 2.0. The not-for-profit organization Eventsourcing, Inc. was recently established. for further development and support of the project.
If you read an article by Fowler or similar sources on the subject of event sourcing, you might have something like this in your brain:
The general idea of this approach is that the user (or any other external system) generates commands, we process them by adding the received events in the event store and updating the "state of the world" in the database, the data from which the user requests.
This approach looks simple and beautiful. We have enough data to “replay” events, we have where to request data on the state of the world from and we can use time-tested databases. On the other hand, I noticed that I wanted a little different from the concept of event sourcing. I wanted to avoid foreseeing the future and this model was somehow not very suitable, because I had to write the updated state to my read-only database.
The very idea that we collect intentions for actions and all the related data leads to the idea that you can always look into the past and use information that was not very useful during the collection of this information (but now there are new observations, new functionality, etc.) What does this mean? Do I have to update my database schema to support new structures? Do I have to replay all events to reload the database?
To be honest, I would like to avoid such difficulties.
During my career I have developed enough projects to understand a few simple truths. One of these truths - the most constant thing - is change. In fact, there is no normal way to find out what your application and data model will look like in a few months. Maybe after the rewriting it was all three times from scratch, and even that is in question.
What if I can postpone guessing the future until I will actually be in this very future? Since I record every event, I can always rebuild the “state of the world”, playing all the old events again, but, as I noted earlier, this is a very expensive pleasure.
But what if I can just send requests to search for events that fall under the necessary criteria, right at the time of the request from the user? For example, suppose we have users and they have email addresses. Suppose we created UserCreated and EmailChanged events. What if, instead of replaying all the events for all users (or even for one!), We just look for UserCreated (id:) and the last EmailChanged (userId:)?
Hm. We can request the necessary information about users only when necessary, without replaying events!
// Pseudocode-ish User's email retrieval
public class User {
public String email() {
return query(equal(EmailChanged.ID, id),
descending(EmailChanged.TIMESTAMP)).first().
email();
}
If the algorithm for finding out the user's email has ever changed, all we need to do is change the User # email () method. Moreover, until I write the User # email () method itself, I don’t need to know that this is exactly how I will structure the data for the presentation. I just have to write down all the data in my events, if possible without losing anything.
More interestingly, this approach leads to the following conclusion: in fact, there is no universal version of the “state of the world”. Just as we humans perceive the world and the reality around us differently, depending on the context, programs can perceive reality through different prisms and I see no reason why they should be limited in this.
Are there any problems with this approach? Of course. For example, if any of your search queries is time consuming. But, when such bottlenecks are detected, they can be optimized through some kind of preprocessing or caching. It would be nice of course to avoid costly requests, but we live in the real world.
Does this mean that we should not have event handlers that execute during event recording? I do not think this is a good way to respond to changes, especially in domains that are implemented by third-party components. I just don’t see the point of doing this for each event.
Here, by the way, is another interesting consequence of using this approach. With the growing popularity of GraphQL as an API, more and more applications are optimizing the amount of data transferred and the number of queries, requesting and receiving only the minimum that they need. Together with the "lazy" collection of data from events, we never build the "state of the world" on those parts of the data that are not requested.
The above approach is used in the Eventsourcing project , including its extension called Domain Protocol. The project also has an adapter for GraphQL . The project is fully open and licensed under Mozilla Public License 2.0. The not-for-profit organization Eventsourcing, Inc. was recently established. for further development and support of the project.