Thoughts on web-API. Part one
At one point, in the process of creating the next web service, I decided to gather all my knowledge and thoughts on the topic of designing web-APIs to serve the needs of client applications and place them in the form of an article or series of articles. Of course, my experience does not claim to be absolute, and constructive criticism and additions are more than welcome.
The reading turned out to be more philosophical than technical, but for fans of the technical part there will be something to reflect on here. I doubt that I will say something fundamentally new in this article, something that you have never heard of, have not read, and about which you yourself did not think. I’ll just try to put everything into a single system, first of all in my own head, and this is already worth a lot. Nevertheless, I will be glad if my thoughts will be useful to you in your practice. So let's go.
Server , in this case we consider an abstract machine in the network, the ability to obtain HTTP-request, process it, and return the correct answer. In the context of this article, its physical essence and internal architecture are absolutely not important, whether it be a student laptop or a huge cluster of industrial servers scattered around the world. To the same extent, it doesn’t matter to you what’s under the hood, who meets a request at the door, Apache or Nginx, which unknown beast, PHP, Python or Ruby processes it and generates a response, which data store is used: Postgresql, MySQL or MongoDB . The main thing is that the server meets the main rule - to hear, understand andforgive to answer.
The client, too, can be anything that can form and send an HTTP request. Until a certain point in this article, we will also not be particularly interested in the goals that the client sets for himself by sending this request, as well as what he will do with the answer. The client may be a JavaScript script running in the browser, a mobile application, an evil (or not so) daemon running on the server, or a too smart refrigerator (there are already some).
For the most part, we will talk about the method of communication between the above two, such a way that they understand each other, and no one has any questions.
REST (Representational state transfer) was originally conceived as a simple and unambiguous interface for data management, involving only a few basic operations with direct network storage (server): data extraction (GET) , saving (POST) , changing (PUT / PATCH) and deleting (DELETE) . Of course, this list has always been accompanied by such options as error handling in the request (whether the request is correctly composed), access control for data (suddenly you should not know this) and validation of incoming data (suddenly you wrote nonsense), in general, with all possible checks which the server fulfills before fulfilling the client’s desire.
In addition, REST has a number of architectural principles, a list of which can be found in any other article on REST. Let's go over them briefly so that they are at hand, and do not have to go anywhere:
Server independence from the client - servers and clients can be instantly replaced by others independently of each other, since the interface between them does not change. The server does not store client states.
Uniqueness of resource addresses - each unit of data (of any degree of nesting) has its own unique URL, which, in essence, is a unique resource identifier.
Example: GET / api / v1 / users / 25 / name
Independence of the data storage format from the transmission format- the server can support several different formats for transferring the same data (JSON, XML etc), but stores the data in its internal format, regardless of the supported ones.
The presence of all the necessary metadata in the response - in addition to the data itself, the server should return request processing details, for example, error messages, various properties of the resource necessary for further work with it, for example, the total number of records in the collection for the correct display of page navigation. We will also go through the varieties of resources.
Classic REST implies that the client works with the server as a flat data warehouse, while nothing is said about the interconnectedness and interdependence of the data among themselves. All this, by default, rests entirely with the client application. However, modern subject areas for which data management systems are being developed, whether social services or Internet marketing systems, imply a complex relationship between entities stored in a database. Support for these relationships, i.e. data integrity, is in the zone of responsibility of the server side, while the client is only an interface for accessing this data. So what are we missing in REST?
The need to distinguish functions as a separate way of data management is dictated by the fact that the principles of encapsulation of methods, atomicity of transactions and maintaining data integrity come into force. In order not to change the data and the relationships between them manually, we simply call the function on the resource (single object or collection) and “feed” it the necessary parameters as an argument. This operation does not fit the REST standards, there is no special verb for it, nor a way to indicate the name of the function, which makes us developers get out of it.
The simplest example is user authorization. We call the function POST / api / v1 / auth / login, we pass it as an argument an object containing the credentials, and in return we get the access key. What is happening with the data on the server side does not bother us.
Another option is to create and break ties between entities. For example, adding a user to a group. We call the POST / api / v1 / groups / 1 / addUser function on the group entity , pass the user object as a parameter, and get the result. An example, of course, is far-fetched, but often when creating connections, additional operations with data on the server side are possible.
And there are also operations that are not directly related to the storage of data as such, for example, sending out notifications, confirming or rejecting any operations (completion of the reporting period etc).
В одной из следующих статей я постараюсь классифицировать эти операции и предложить варианты возможных запросов и ответов, основываясь на том, с какими из них мне приходилось сталкиваться на практике.
It often happens, and client developers will understand what I mean, that it is more convenient for a client application to create / modify / delete / several homogeneous objects at once with one request, and each object can have its own server side verdict. There are at least a few options: either all the changes have been completed, or they have been partially implemented (for some objects), or an error has occurred. Well, there are also several strategies: apply changes only in case of success for everyone, either apply partially, or roll back in case of any error, and this already draws to a full-fledged transaction mechanism.
For web-api, striving for the ideal, I would also like to somehow bring such operations to the system. I will try to do this in one of the sequels.
It often happens that based on the data stored on the server, we need to get statistical squeezing or data formatted in a special way: for example, to build a graph on the client side. In fact, this is data generated on demand, to one degree or another on the fly, and read-only, so it makes sense to put them in a separate category. One of the distinguishing features of statistics, in my opinion, is that they do not have a unique ID.
I am sure that this is far from everything that you may encounter when developing real applications, and I will be glad to see your additions and corrections.
The key data type in communication between the client and server is the object. In essence, an object is a list of properties and their corresponding values. We can send the object to the server in the request and get the result of the request as an object. Moreover, the object will not necessarily be a real entity stored in the database, at least in the form in which it is sent or received. For example, credentials for authorization are transmitted as an object, but they are not an independent entity. Even objects stored in the database tend to acquire additional intrasystem-specific properties, for example, creation and editing dates, various system labels and flags. Properties of objects can be either their own scalar values or contain related objects and collections of objects that are not part of the object. Part of the properties of objects can be editable, part of the system, read-only, and part can be statistical in nature and calculated on the fly (for example, the number of likes). Some properties of the object may be hidden, depending on the rights of the user.
Speaking of collections, we mean a kind of server resource that allows you to work with a list of homogeneous objects, i.e. add, delete, modify objects and select from them. In addition, the collection theoretically may have its own properties (for example, the maximum number of elements per page) and functions (here I am confused, but this also happened).
In its pure form, scalar values as a separate entity in my memory were extremely rare. Usually they figured as properties of objects or collections, and as such they may be available both for reading and writing. For example, a username can be obtained and changed individually GET / api / v1 / users / 1 / name . In practice, this feature rarely comes in handy, but if necessary, I would like it to be at hand. This is especially true of the properties of the collection, for example, the number of posts (with or without filtering): GET / api / v1 / news / count .
Files are files - they should be considered as a single indivisible unit. Another question is that in most cases when saving a file in the database, a service entity can be created that contains metadata of this file: size, real name, status etc.
To be continued...
The reading turned out to be more philosophical than technical, but for fans of the technical part there will be something to reflect on here. I doubt that I will say something fundamentally new in this article, something that you have never heard of, have not read, and about which you yourself did not think. I’ll just try to put everything into a single system, first of all in my own head, and this is already worth a lot. Nevertheless, I will be glad if my thoughts will be useful to you in your practice. So let's go.
Approach One: Actors
Client and server
Server , in this case we consider an abstract machine in the network, the ability to obtain HTTP-request, process it, and return the correct answer. In the context of this article, its physical essence and internal architecture are absolutely not important, whether it be a student laptop or a huge cluster of industrial servers scattered around the world. To the same extent, it doesn’t matter to you what’s under the hood, who meets a request at the door, Apache or Nginx, which unknown beast, PHP, Python or Ruby processes it and generates a response, which data store is used: Postgresql, MySQL or MongoDB . The main thing is that the server meets the main rule - to hear, understand and
The client, too, can be anything that can form and send an HTTP request. Until a certain point in this article, we will also not be particularly interested in the goals that the client sets for himself by sending this request, as well as what he will do with the answer. The client may be a JavaScript script running in the browser, a mobile application, an evil (or not so) daemon running on the server, or a too smart refrigerator (there are already some).
For the most part, we will talk about the method of communication between the above two, such a way that they understand each other, and no one has any questions.
REST Philosophy
REST (Representational state transfer) was originally conceived as a simple and unambiguous interface for data management, involving only a few basic operations with direct network storage (server): data extraction (GET) , saving (POST) , changing (PUT / PATCH) and deleting (DELETE) . Of course, this list has always been accompanied by such options as error handling in the request (whether the request is correctly composed), access control for data (suddenly you should not know this) and validation of incoming data (suddenly you wrote nonsense), in general, with all possible checks which the server fulfills before fulfilling the client’s desire.
In addition, REST has a number of architectural principles, a list of which can be found in any other article on REST. Let's go over them briefly so that they are at hand, and do not have to go anywhere:
Server independence from the client - servers and clients can be instantly replaced by others independently of each other, since the interface between them does not change. The server does not store client states.
Uniqueness of resource addresses - each unit of data (of any degree of nesting) has its own unique URL, which, in essence, is a unique resource identifier.
Example: GET / api / v1 / users / 25 / name
Independence of the data storage format from the transmission format- the server can support several different formats for transferring the same data (JSON, XML etc), but stores the data in its internal format, regardless of the supported ones.
The presence of all the necessary metadata in the response - in addition to the data itself, the server should return request processing details, for example, error messages, various properties of the resource necessary for further work with it, for example, the total number of records in the collection for the correct display of page navigation. We will also go through the varieties of resources.
What we lack
Classic REST implies that the client works with the server as a flat data warehouse, while nothing is said about the interconnectedness and interdependence of the data among themselves. All this, by default, rests entirely with the client application. However, modern subject areas for which data management systems are being developed, whether social services or Internet marketing systems, imply a complex relationship between entities stored in a database. Support for these relationships, i.e. data integrity, is in the zone of responsibility of the server side, while the client is only an interface for accessing this data. So what are we missing in REST?
Function calls
The need to distinguish functions as a separate way of data management is dictated by the fact that the principles of encapsulation of methods, atomicity of transactions and maintaining data integrity come into force. In order not to change the data and the relationships between them manually, we simply call the function on the resource (single object or collection) and “feed” it the necessary parameters as an argument. This operation does not fit the REST standards, there is no special verb for it, nor a way to indicate the name of the function, which makes us developers get out of it.
The simplest example is user authorization. We call the function POST / api / v1 / auth / login, we pass it as an argument an object containing the credentials, and in return we get the access key. What is happening with the data on the server side does not bother us.
Another option is to create and break ties between entities. For example, adding a user to a group. We call the POST / api / v1 / groups / 1 / addUser function on the group entity , pass the user object as a parameter, and get the result. An example, of course, is far-fetched, but often when creating connections, additional operations with data on the server side are possible.
And there are also operations that are not directly related to the storage of data as such, for example, sending out notifications, confirming or rejecting any operations (completion of the reporting period etc).
В одной из следующих статей я постараюсь классифицировать эти операции и предложить варианты возможных запросов и ответов, основываясь на том, с какими из них мне приходилось сталкиваться на практике.
Множественные операции
It often happens, and client developers will understand what I mean, that it is more convenient for a client application to create / modify / delete / several homogeneous objects at once with one request, and each object can have its own server side verdict. There are at least a few options: either all the changes have been completed, or they have been partially implemented (for some objects), or an error has occurred. Well, there are also several strategies: apply changes only in case of success for everyone, either apply partially, or roll back in case of any error, and this already draws to a full-fledged transaction mechanism.
For web-api, striving for the ideal, I would also like to somehow bring such operations to the system. I will try to do this in one of the sequels.
Statistical queries, aggregators, data formatting
It often happens that based on the data stored on the server, we need to get statistical squeezing or data formatted in a special way: for example, to build a graph on the client side. In fact, this is data generated on demand, to one degree or another on the fly, and read-only, so it makes sense to put them in a separate category. One of the distinguishing features of statistics, in my opinion, is that they do not have a unique ID.
I am sure that this is far from everything that you may encounter when developing real applications, and I will be glad to see your additions and corrections.
Varieties of data
The objects
The key data type in communication between the client and server is the object. In essence, an object is a list of properties and their corresponding values. We can send the object to the server in the request and get the result of the request as an object. Moreover, the object will not necessarily be a real entity stored in the database, at least in the form in which it is sent or received. For example, credentials for authorization are transmitted as an object, but they are not an independent entity. Even objects stored in the database tend to acquire additional intrasystem-specific properties, for example, creation and editing dates, various system labels and flags. Properties of objects can be either their own scalar values or contain related objects and collections of objects that are not part of the object. Part of the properties of objects can be editable, part of the system, read-only, and part can be statistical in nature and calculated on the fly (for example, the number of likes). Some properties of the object may be hidden, depending on the rights of the user.
Object Collections
Speaking of collections, we mean a kind of server resource that allows you to work with a list of homogeneous objects, i.e. add, delete, modify objects and select from them. In addition, the collection theoretically may have its own properties (for example, the maximum number of elements per page) and functions (here I am confused, but this also happened).
Scalar values
In its pure form, scalar values as a separate entity in my memory were extremely rare. Usually they figured as properties of objects or collections, and as such they may be available both for reading and writing. For example, a username can be obtained and changed individually GET / api / v1 / users / 1 / name . In practice, this feature rarely comes in handy, but if necessary, I would like it to be at hand. This is especially true of the properties of the collection, for example, the number of posts (with or without filtering): GET / api / v1 / news / count .
Files
Files are files - they should be considered as a single indivisible unit. Another question is that in most cases when saving a file in the database, a service entity can be created that contains metadata of this file: size, real name, status etc.
To be continued...