August 31, 2017 at 12:52Stripe API version control system as a separate tool
- Transfer
The author of the article talks about the device version control system, which is implemented at Stripe.
When it comes to APIs, change is an unpopular thing. While many software developers are accustomed to working in frequent and fast iterations, API developers lose this flexibility as soon as they get at least the first user of their interface. Many of us are familiar with the history of the evolution of the Unix operating system.
In 1994, the book “Unix-Haters Handbook” was published, which touched on a whole list of various hot topics, from the names of teams optimized for teletypes with a completely incomprehensible history of origin to the irreversible deletion of data and obscure programs from a lot of options. More than 20 years later, the vast majority of these complaints are still relevant despite the variety of legacy systems and branches available today. Unix has become so widely popular that changing its behavior can lead to far-reaching consequences. For better or worse, certain agreements have already been formed between him and his users that determine the behavior of Unix interfaces.
In a similar way, the API is a communication contract that cannot be changed without a significant share of cooperation and effort on both sides. Many businesses rely on Stripe as an infrastructure provider and therefore we have been thinking about this type of interaction from the very beginning of our company. Currently, we have managed to maintain support for each version of our API since the company appeared in 2011. In this article, we would like to share with you how we at Stripe manage to organize work with API versions.
The code written for integration with the API is initially fraught with some expectations. If the endpoint returns a verified boolean field to indicate the status of the bank account, the user can write something like this:
If after that we replace the boolean field verified with a status field, which may include the verified field (as we did in 2014), the code will stop working because it depends on a field that no longer exists. This type of change leads to backward incompatibility and therefore we avoid such changes. Fields that were present earlier should continue to be present and always keep the same type and name. However, not all changes lead to backward incompatibility. For example, it is safe to add a new API endpoint or a completely new field to an existing point.
With sufficient coordination of efforts, we could keep users up to date with upcoming changes and ask us to update our integrations in advance, but even if this were possible, this approach cannot be called customer-oriented. Like connecting to a power grid or water supply, our API should work as long as possible without any need to make changes.
Stripe provides the economic infrastructure for the Internet. As electricity producers do not need to change voltage every two years, we also believe that our users must be sure that the API will remain stable for as long as possible.
There is a common approach that allows you to develop an Internet API - support for different versions. When making requests, users indicate the version of the API, and its suppliers can make changes in its next iteration, while maintaining compatibility in the current one. As new versions are released, users can switch to newer ones at a convenient time for them.
The essence of the most common version control scheme today is to use names like v1, v2 and v3, transmitted as a prefix to the URL (for example, / v1 / widgets) or through an HTTP header like Accept. This approach may work, but its main drawback is that when the size of the update between versions is large, and it itself involves serious changes, that in terms of complexity of the transition, it is equivalent to the need to re-integrate from scratch.
The positive aspects of such a transition are not so obvious, since there is always a class of users who cannot or do not want to upgrade, as a result of which they are trapped in older versions of the API. In this case, suppliers face a difficult choice between abandonment of old versions and, as a result, loss of such customers, and continued support of old versions forever, which entails significant costs. Despite the fact that the second option may, at first glance, seem to be the right decision from the point of view of client-oriented, support for outdated versions indirectly affects the quality of the project as a whole, since in fact it reduces the pace of work on innovations. Instead of developing new features, engineers’s working hours are partially eaten up by the support of old code.
We at Stripe apply version control named after the release date (e.g. 2017-05-24). Despite their backward incompatibility, each such update contains a small set of changes that make updating and updating its integration a smooth and relatively simple process.
The first time the user accesses the API, their account is automatically assigned the latest version available, after which the system automatically implies that each subsequent API call will refer to this version. This approach eliminates the situation when users accidentally receive a change that violates their integration, and also makes the initial integration less painful by reducing the amount of work to configure it. Users can force a version of each individual request by manually setting the Stripe-Version header, or by updating the version in their account from the Stripe control panel.
Some readers may already have noticed that the Stripe API also defines major versions using a path prefix (e.g. / v1 / charges). And although we reserve the right to use this scheme sooner or later, it is unlikely that this will happen in the foreseeable future. As noted above, major changes, as a rule, turn upgrades into a difficult and unpleasant task, and it’s hard for us to imagine such an important API redesign that could justify such inconvenience to users. Our current approach has been effective for nearly a hundred backward-incompatible updates released over the past six years.
Versioning is always a compromise between improving developer tools and the additional burden of supporting older versions. We strive in every possible way to achieve the first, while minimizing the cost of work on the second item. To achieve these goals, we have introduced a version control system. Let's look at a small example of how it works. Each possible answer from the Stripe API is written in the form of a class called the API resource. Such resources define their possible fields using a subject-oriented language:
API resources are written in such a way that the structure they describe is what we expect to receive from the current version of the API. When we need to make a back-incompatible change, we encapsulate it in the version change module, which defines the documentation for the change, the conversion itself, and the set of types of API resources that fall under the change:
In other cases, changes are assigned in accordance with the data from the master list:
Version changes are written so that, if necessary, they are automatically applied in the reverse order starting from the current version of the API. Each version change assumes, despite the presence of newer changes ahead, that the data they receive will look the same as they were originally written.
When generating an answer, the API primarily formats the data by describing the resource API of the current version. After that comes the definition of the target version of the API based on:
After that, the API does a backward crawl of the version and applies each version change module in its path until it reaches the desired version.
Before the API returns a response, all requests are processed by the version change modules.
Version change modules allow you to abstract from older versions of the API when working with core code. As a result, most of the time developers can avoid thinking about older versions while developing new products.
Most of our backward-incompatible API changes alter its response, but this happens forever. Sometimes a more complex change is required, which follows from the module that defines it. We assign the has_side_effects note to such modules (there are side effects) and the transformation they describe turns into an idle code:
The fact of their deactivation will also be checked in other parts of the code:
This reduction in encapsulation makes it difficult to support changes with side effects, so we try to avoid this approach.
One of the benefits of standalone version change modules is that they can declare documentation describing which fields and resources they affect. We can use this to quickly provide our users with more useful information. For example, the change log of our API is generated programmatically and updated as soon as we deploy new versions of services.
We also adapt the API reference documentation to the needs of individual users. She checks that the user is authorized in the system and leaves notes for the fields, based on the current version of the API of his account. In the image below, for example, we warn the developer that backward incompatible changes have been made in newer versions of the API compared to its fixed version. The request field of the event was previously a string, but now it is a subobject that also contains the idempotency key (version changes created in the framework of the code shown above):
Our documentation defines the user version of the API and shows the corresponding warnings.
Providing enhanced backward compatibility is not in vain. Each new version adds more code that needs to be understood and maintained. We try to write as cleanly as possible, but over time, dozens of checks for version changes that are not clearly encapsulated can lead to the project becoming overgrown with unnecessary things and as a result it will become slower, more fragile and its readability will be lost. In order to avoid the accumulation of this kind of expensive technical debt, we took several measures.
Despite the fact that we have a well-thought-out version control system, we are doing everything possible to avoid its use and, above all, we are trying to build the architecture of our API from the very beginning. Any planned changes go through a simple review process, in which they are described in a brief information document and sent by mailing list. This allows you to look at the proposed changes more broadly, from the point of view of different departments of the company. The likelihood of detecting errors and inconsistencies increases before they get into the release.
We always try to remember the balance between the need to maintain previous versions and the development of new features. Compatibility support is important, but even so, we expect that in the end we will begin to abandon the old versions. Helping users to upgrade to new versions gives them access to new features and simplifies the foundation we use to create new features.
The combination of rolling out the version and the internal framework that supports them allowed us to seriously expand our user base and make a huge number of changes to the API, which, however, had practically no effect on the quality of existing integrations. This approach is based on several principles that we have chosen as a result of many years of practice. We believe that it is important for API updates to meet the following criteria:
Despite the fact that we are very interested in observing disputes and developments in topics such as REST vs. GraphQL vs. gRPC, as well as - in a broader sense - discussions of how the API will look in the future, we expect to continue supporting version control schemes for quite some time.
When it comes to APIs, change is an unpopular thing. While many software developers are accustomed to working in frequent and fast iterations, API developers lose this flexibility as soon as they get at least the first user of their interface. Many of us are familiar with the history of the evolution of the Unix operating system.
In 1994, the book “Unix-Haters Handbook” was published, which touched on a whole list of various hot topics, from the names of teams optimized for teletypes with a completely incomprehensible history of origin to the irreversible deletion of data and obscure programs from a lot of options. More than 20 years later, the vast majority of these complaints are still relevant despite the variety of legacy systems and branches available today. Unix has become so widely popular that changing its behavior can lead to far-reaching consequences. For better or worse, certain agreements have already been formed between him and his users that determine the behavior of Unix interfaces.
In a similar way, the API is a communication contract that cannot be changed without a significant share of cooperation and effort on both sides. Many businesses rely on Stripe as an infrastructure provider and therefore we have been thinking about this type of interaction from the very beginning of our company. Currently, we have managed to maintain support for each version of our API since the company appeared in 2011. In this article, we would like to share with you how we at Stripe manage to organize work with API versions.
The code written for integration with the API is initially fraught with some expectations. If the endpoint returns a verified boolean field to indicate the status of the bank account, the user can write something like this:
if bank_account[:verified]
...
else
...
EndIf after that we replace the boolean field verified with a status field, which may include the verified field (as we did in 2014), the code will stop working because it depends on a field that no longer exists. This type of change leads to backward incompatibility and therefore we avoid such changes. Fields that were present earlier should continue to be present and always keep the same type and name. However, not all changes lead to backward incompatibility. For example, it is safe to add a new API endpoint or a completely new field to an existing point.
With sufficient coordination of efforts, we could keep users up to date with upcoming changes and ask us to update our integrations in advance, but even if this were possible, this approach cannot be called customer-oriented. Like connecting to a power grid or water supply, our API should work as long as possible without any need to make changes.
Stripe provides the economic infrastructure for the Internet. As electricity producers do not need to change voltage every two years, we also believe that our users must be sure that the API will remain stable for as long as possible.
API Versioning Schemes
There is a common approach that allows you to develop an Internet API - support for different versions. When making requests, users indicate the version of the API, and its suppliers can make changes in its next iteration, while maintaining compatibility in the current one. As new versions are released, users can switch to newer ones at a convenient time for them.
The essence of the most common version control scheme today is to use names like v1, v2 and v3, transmitted as a prefix to the URL (for example, / v1 / widgets) or through an HTTP header like Accept. This approach may work, but its main drawback is that when the size of the update between versions is large, and it itself involves serious changes, that in terms of complexity of the transition, it is equivalent to the need to re-integrate from scratch.
The positive aspects of such a transition are not so obvious, since there is always a class of users who cannot or do not want to upgrade, as a result of which they are trapped in older versions of the API. In this case, suppliers face a difficult choice between abandonment of old versions and, as a result, loss of such customers, and continued support of old versions forever, which entails significant costs. Despite the fact that the second option may, at first glance, seem to be the right decision from the point of view of client-oriented, support for outdated versions indirectly affects the quality of the project as a whole, since in fact it reduces the pace of work on innovations. Instead of developing new features, engineers’s working hours are partially eaten up by the support of old code.
We at Stripe apply version control named after the release date (e.g. 2017-05-24). Despite their backward incompatibility, each such update contains a small set of changes that make updating and updating its integration a smooth and relatively simple process.
The first time the user accesses the API, their account is automatically assigned the latest version available, after which the system automatically implies that each subsequent API call will refer to this version. This approach eliminates the situation when users accidentally receive a change that violates their integration, and also makes the initial integration less painful by reducing the amount of work to configure it. Users can force a version of each individual request by manually setting the Stripe-Version header, or by updating the version in their account from the Stripe control panel.
Some readers may already have noticed that the Stripe API also defines major versions using a path prefix (e.g. / v1 / charges). And although we reserve the right to use this scheme sooner or later, it is unlikely that this will happen in the foreseeable future. As noted above, major changes, as a rule, turn upgrades into a difficult and unpleasant task, and it’s hard for us to imagine such an important API redesign that could justify such inconvenience to users. Our current approach has been effective for nearly a hundred backward-incompatible updates released over the past six years.
Under the hood of a version control system
Versioning is always a compromise between improving developer tools and the additional burden of supporting older versions. We strive in every possible way to achieve the first, while minimizing the cost of work on the second item. To achieve these goals, we have introduced a version control system. Let's look at a small example of how it works. Each possible answer from the Stripe API is written in the form of a class called the API resource. Such resources define their possible fields using a subject-oriented language:
class ChargeAPIResource
required :id, String
required :amount, Integer
EndAPI resources are written in such a way that the structure they describe is what we expect to receive from the current version of the API. When we need to make a back-incompatible change, we encapsulate it in the version change module, which defines the documentation for the change, the conversion itself, and the set of types of API resources that fall under the change:
class CollapseEventRequest < AbstractVersionChange
description \
“Cобытийные объекты (и веб-хуки) теперь генерируют “ \
“подобъект request, содержащий id запроса и “ \
“ключ идемпотентности вместо одной строки “ \
“с id запроса.”
response EventAPIResource do
change :request, type_old: String, type_new: Hash
run do |data|
data.merge(:request => data[:request][:id])
end
end
end
In other cases, changes are assigned in accordance with the data from the master list:
class VersionChanges
VERSIONS = {
'2017-05-25' => [
Change::AccountTypes,
Change::CollapseEventRequest,
Change::EventAccountToUserID
],
'2017-04-06' => [Change::LegacyTransfers],
'2017-02-14' => [
Change::AutoexpandChargeDispute,
Change::AutoexpandChargeRule
],
'2017-01-27' => [Change::SourcedTransfersOnBts],
...
}
endVersion changes are written so that, if necessary, they are automatically applied in the reverse order starting from the current version of the API. Each version change assumes, despite the presence of newer changes ahead, that the data they receive will look the same as they were originally written.
When generating an answer, the API primarily formats the data by describing the resource API of the current version. After that comes the definition of the target version of the API based on:
- The Stripe-Version header, if any.
- The version of the authorized OAuth application, if the request is made on behalf of the user.
- The version assigned to the user that is assigned when the very first request is sent to Stripe.
After that, the API does a backward crawl of the version and applies each version change module in its path until it reaches the desired version.
Before the API returns a response, all requests are processed by the version change modules.
Version change modules allow you to abstract from older versions of the API when working with core code. As a result, most of the time developers can avoid thinking about older versions while developing new products.
Side Effects Changes
Most of our backward-incompatible API changes alter its response, but this happens forever. Sometimes a more complex change is required, which follows from the module that defines it. We assign the has_side_effects note to such modules (there are side effects) and the transformation they describe turns into an idle code:
class LegacyTransfers < AbstractVersionChange
description "..."
has_side_effects
EndThe fact of their deactivation will also be checked in other parts of the code:
VersionChanges.active?(LegacyTransfers)This reduction in encapsulation makes it difficult to support changes with side effects, so we try to avoid this approach.
Declarative changes
One of the benefits of standalone version change modules is that they can declare documentation describing which fields and resources they affect. We can use this to quickly provide our users with more useful information. For example, the change log of our API is generated programmatically and updated as soon as we deploy new versions of services.
We also adapt the API reference documentation to the needs of individual users. She checks that the user is authorized in the system and leaves notes for the fields, based on the current version of the API of his account. In the image below, for example, we warn the developer that backward incompatible changes have been made in newer versions of the API compared to its fixed version. The request field of the event was previously a string, but now it is a subobject that also contains the idempotency key (version changes created in the framework of the code shown above):
Our documentation defines the user version of the API and shows the corresponding warnings.
Minimizing change
Providing enhanced backward compatibility is not in vain. Each new version adds more code that needs to be understood and maintained. We try to write as cleanly as possible, but over time, dozens of checks for version changes that are not clearly encapsulated can lead to the project becoming overgrown with unnecessary things and as a result it will become slower, more fragile and its readability will be lost. In order to avoid the accumulation of this kind of expensive technical debt, we took several measures.
Despite the fact that we have a well-thought-out version control system, we are doing everything possible to avoid its use and, above all, we are trying to build the architecture of our API from the very beginning. Any planned changes go through a simple review process, in which they are described in a brief information document and sent by mailing list. This allows you to look at the proposed changes more broadly, from the point of view of different departments of the company. The likelihood of detecting errors and inconsistencies increases before they get into the release.
We always try to remember the balance between the need to maintain previous versions and the development of new features. Compatibility support is important, but even so, we expect that in the end we will begin to abandon the old versions. Helping users to upgrade to new versions gives them access to new features and simplifies the foundation we use to create new features.
Principles underlying change
The combination of rolling out the version and the internal framework that supports them allowed us to seriously expand our user base and make a huge number of changes to the API, which, however, had practically no effect on the quality of existing integrations. This approach is based on several principles that we have chosen as a result of many years of practice. We believe that it is important for API updates to meet the following criteria:
- Lightness . Upgrades should be made as low-cost (both for users and for us) as possible.
- First-class approach . Turn version control into a first-class concept for your API. Make it so that you can use it for accurate documentation and improvement of tools, as well as for automatic generation of changelogs.
- Fixed cost . Make sure that older versions add only a minimum of support work, by firmly encapsulating them in version change modules. In other words, the less you need to think about old behaviors while writing new code, the better.
Despite the fact that we are very interested in observing disputes and developments in topics such as REST vs. GraphQL vs. gRPC, as well as - in a broader sense - discussions of how the API will look in the future, we expect to continue supporting version control schemes for quite some time.