From API first on Swagger to Single contract on RAML

image

Hi% username%!

You probably know what API interfaces are and how much depends on them in your project. Moreover, I also believe that you are already familiar with the fact that API is the first approach and you know that Swagger and its Open API are among the most popular tools to help it follow.

But in this article I want to tell you about the approach to the implementation of the API first, which is conceptually different from what Swagger and Apiary offers. At the head of the idea is the concept of Single contract and the possibility of its implementation based on RAML 1.0.

Under the cut:

  • A brief description of the principles of the API first;
  • Single contract - introduction of the concept, prerequisites for the appearance, consideration of the possibility of its implementation on the basis of OAS (Swagger);
  • RAML + annotations + overlays as a base for Single contract , examples;
  • RAML problems, conceptual differences of developers;
  • The idea of ​​a SaaS service based on the above idea (picture of the prototype above).



From API first on Swagger to Single contract on RAML


When designing modern software systems, it often becomes necessary to coordinate and develop interfaces for the interaction of their components with each other. In the last decade, SPA and thick mobile applications interacting with the server through API interfaces have gained immense popularity and development. If earlier the development of an interactive web site was done by phasing the server side code to generate HTML markup and then transferring it to the browser of the client, now the development of dynamic web applications has shifted towards the creation of a single service API and parallel development of many applications (including SPA) running with this API as with the main data source. This approach allows you to more conveniently divide tasks, organize teams,

Such a single point of communication requires a formal and unambiguous definition, this document is the API specification. Various technologies and languages ​​are used to develop and document API specifications today, for example: OAS (Swagger), Apiary and RAML.

The following three items define the nature of the API first approach:

  1. The API interface should be the very first client interface of the application being developed;
  2. The API specification is first developed, and then the software part of its clients;
  3. The life stages of an API interface must match the life stages of its documentation.

If we consider the process based on the foregoing, the API specification lies at the center of the development process, and all nodes that make up the system and use this API as an interaction gateway are clients of the API specification. Thus, the server part of the system can be considered the same API client of the specification, like any other node that uses the API to communicate with it. Domain domain application models do not have to match the models described in the API specification. Their possible intentional coincidences with class structures in the code of client applications or database schema structures are introduced rather to simplify the development process, for example, when using a code generator according to the OAS specification. Logically, the above can be brought under the definition of Single contract .Single contract - many clients.

Single Contract. Contract Tools and Libraries


The term Single contract does not claim any participation in criticism for its use in the text of the article. Its use, in this context, is my personal idea.
Expanding the concept of API first , to the more general Single contract, allows us to consider the API specification not only as a formal description of the interaction interface between system components, but also as a single contract used by any number of external libraries and tools as a configuration source. In this case, these tools and libraries can be perceived as contract clients along with SPA or mobile applications. Examples of such customers include:

  • Documentation Generator
  • Mock-server API
  • Load Testing Service
  • Requests / Responses Validation Library
  • Code generator
  • UI Interface Generator
  • etc.

Single contract for such clients is a single configuration file and data source. Contract tools work only on the basis of information obtained from a particular contract. Obviously, for a full-fledged functional of such heterogeneous clients as a mock server API, a single API description is not enough, additional meta information is needed, for example, a description of the connections between the GET request parameters (resource id) and the data that the server should return, hints indicating the response fields and query parameters used to organize pagination. Further, this example will be discussed in more detail. Specific information for specific instruments, at the same time, must exist and be maintained inextricably from the main document, otherwise it will lead to a violation of the concept of a single contract.

Swagger (OAS) as a Single contract description tool


The existing most popular on the market, Swagger (OAS) and Apiary (Blueprint) allow you to describe HTTP API interfaces using special languages: Open API based on YAML or JSON, Blueprint based on Markdown, which makes the specifications easy to read. There are also many tools and libraries created by the large open-source community. Swagger is currently widely distributed and, one might say, has become the de facto API standard first. Many external systems support importing Swagger specifications, such as SoapUI , Readme.io , Apigee , etc. In addition, the existing SaaS Swagger Hub and Apiaryallow users to create projects, upload or create their own specifications, use built-in documentation generators and mock servers, as well as publish links to access them from outside.

Swagger along with its OAS 3.0 look pretty confident and its functionality to describe the API (especially simple) is enough in most cases. The following is a list of pros and cons of Swagger:

Pros:

  • Clear and easy to read description language;
  • Large open-source community;
  • Many official and open-source editors, generators, libraries;
  • Having a core development team constantly working to develop and improve the format;
  • Conditionally free hub for specifications;
  • Detailed official documentation;
  • Low threshold of entry.

Minuses:

  • Weak modularity support;
  • Absence of autogeneration of sample response requests based on the description of their structures;
  • There are often problems with the poor stability of SmartBear products (the authors of the swagger) and the late reaction of the developer to this (the opinion is based purely on personal experience of use, and on the experience of our team).

But the main limitation that prevents using OAS as a Single contract description tool is the inability to attach custom meta information to describe additional parameters of target tools / libraries.
Therefore, all tools that work on the basis of Swagger-specifications must be content with the set of information that can accommodate the basic format.

For example, the implementation of a smart mock api server requires more information than a specification document can provide, which is why the built-in Swagger Hub mock API is only capable of generating fake data based on data types / structures obtained from the specification document. Undoubtedly, this is not enough and such functionality of the mock-server can be satisfied only by a simple API client.

In our company, during the development of one of the projects (React SPA + API server), the following mock-server functionality was required:

  • imitation of pagination. The server should not return completely random values ​​of the currentPage, nextPage, pagesTotal fields in response to list requests, but be able to simulate the actual behavior of the pagination mechanism with the generation of values ​​of these metafields depending on the received page value from the client;
  • generation of response bodies containing different data sets depending on the specific parameter of the incoming request;
  • the ability to build real connections between fake objects: the foo_id field of the Bar entity must refer to the previously generated Foo entity. This can be achieved by adding the support of idempotency mock-server;
  • Imitation of work of various authorization methods: OAuth2, JWT, etc.

Without all this, it is very difficult to develop a SPA in parallel with the development of the server part of the system. And, at the same time, such a mock server, due to the previously described, is almost impossible to implement without additional specific meta information that could be stored directly in the API specification and inform it about the required behavior when imitating the next endpoint. This problem can be solved by adding the required parameters as a separate file with configurations parallel to the base OAS specification, but in this case, you will need to support these two different sources separately.

If there are more than one mock server operating in the environment of the development process according to this principle, we will get a “zoo” of tools, each of which, if it has its own unique functionality, is forced to have its own unique configuration file, logically linked to the base API -specifications, but actually located separately and living “its own life”.

image

Problem: the developer will be forced to keep all configurations up-to-date after changing versions of the base specification, often in completely different places and formats.

Some examples of services that work on a similar principle:

  • SoapUI is a system for testing REST & SOAP interfaces. Supports project import from Swagger-specification. When the Swagger base specification is changed, the configuration of the draft based on the call API API continues to exist in parallel and requires manual synchronization;
  • Other SmartBear Products ;
  • Apigee is a lifecycle management API service. Uses Swagger-specifications as templates, based on which, allows you to initialize your configuration of internal services. Automatic synchronization is also missing;
  • Readme.io is a service that allows you to create beautiful documentation based on the Swagger-specification, and also has a mechanism for tracking changes in the basic specification and resolving conflicts by updating the project configuration on the service side. Surely, this required the excessive complexity of the development of this service.

You can add many other services to this list that provide the integration feature with the Swagger specification. Integration for most of them means the usual copying of the basic structure of the Swagger-specification and the subsequent auto-completion of the local configuration fields without synchronization with changes in the basic specification.

RAML, annotations, overlays


The desire to find a tool that eliminates the previously mentioned OAS restriction, which makes it possible to view the specification as a single contract for all client tools, led us to become familiar with the RAML language. About RAML is written enough, you can read, for example, https://www.infoq.com/articles/power-of-raml . RAML developers have tried to lay in the language support for modularity at the level of its concept. Now each company or individual developer can create their own or use ready-made public dictionaries when designing an API, redefine and inherit ready-made data models. Starting from version 1.0, RAML supports 5 different types of external modules: include, library, extension, trait, overlaythat allows you to apply each of them depending on the task as flexibly as possible.

The time has come to discuss the main possibility of RAML, which, for reasons not entirely clear, has no analogues in OAS and Blueprint - Annotations.
RAML Annotations is the ability to attach custom metadata to basic language structures.
It is this function RAML and became the reason for writing this article.

Example:

#%RAML1.0
title: ExampleAPI
mediaType: application/json
# Annotation types block may be placed into external file
annotationTypes:
  validation-rules:
    description: |
      Describes strict validation rules for the model properties.
      Can be used by validation library
    allowedTargets: [ TypeDeclaration ]
    type: string[]
  info-tip:
    description: |
      Can be used by Documentation generator for showing tips
    allowedTargets: [ Method, DocumentationItem, TypeDeclaration ]
    type: string
  condition:
    description: |
      Named example can be returned if condition is evaluated to true.
      Can be used by Intelligent mock server
    allowedTargets: [ Example ]
    type: string
types:
  Article:
    type: object
    properties:
      id:
        type: integer
      title: string
      paragraphs: Paragraph[]
      createdAt:
        type: string
        (validation-rules): ["regex:/\d{4}-[01]\d-[0-3]\dT[0-2]\d:[0-5]\d:[0-5]\d(?:\.\d+)?Z?/"]
  Paragraph:
    type: object
    properties:
      order:
        type: integer
        (validation-rules): ["min:0"]
      content: string
        (validation-rules): ["max-length:1024"]
/articles/{articleId}:
  get:
    (info-tip): This endpoint is deprecated
    description: ReturnsArticleobjectbyID
    responses:
      200:
        body:
          application/json:
            type: Article

User annotation structures themselves must have clear RAML descriptions. For this, a special section annotationTypes is used , from which definitions can also be transferred to an external module. Thus, it becomes possible to determine the special parameters of an external tool in the form of annotations associated with the basic definition of the RAML API. In order to avoid cluttering up the base specification with a huge number of annotations for various external tools, there is support for the possibility of their removal into separate files - overlays (as well as possible in extensions ), with a classification by application. Here is what is said about overlays in the documentation for RAML ( https://github.com/raml-org/raml-spec/blob/master/versions/raml-10/raml-10.md#overlays):
Analogue of the RAML API definition while preserving its behavioral, functional aspects. Definition of the APIs: its resources, methods, parameters, bodies, responses, and so on. These nodes cannot be changed by applying an overlay. In contrast, other nodes, such as those that address the functional interface, or These nodes can be changed by applying an overlay.

Overlays are particularly important for separating interface from implementation. Overlays enable you to be controlled tightly, such as a contracting different paces. For example, it can be used to provide an example of the APIs. These things can be controlled through a rigorous version and change management process.
In other words, this functionality allows you to “separate the wheat from the chaff,” for example, the basic description of the API specification, from additional meta-information specific to a particular tool using it for work. Meta-information in each individual overlay is “hung” on various specification blocks in the form of annotations.

An example of a basic structure:

#%RAML1.0
title: PhrasesAPI
mediaType: application/json
types:
  Phrase:
    type: object
    properties:
      content: string
/phrases:
  get:
    queryParameters:
      whoSaid: string
    responses:
      200:
        body:
          application/json:
            type: Phrase

Overlay:
#%RAML 1.0 Overlay
usage: Applies annotations for Intelligent mock server
extends: example_for_article_2_1.raml
annotationTypes:
  condition:
    description: |
      Named example
      can be returned
      if condition is
      evaluated totrue
    type: string
    allowedTargets: Example
/phrases:
  get:
    responses:
      200:
        body:
          application/json:
            examples:
              firstExample:
                (condition): $whoSaid is Hamlet
                content: "To be, or not to be?"
              secondExample:
                (condition): $whoSaid is Homer Simpson
                content: "D'oh!"

As a result, it becomes possible to implement a single contract: all the functional, behavioral and meta information is stored and versioned in one single place, and the contractual tools - the contract clients must have the support of the annotations used in this specification. On the other hand, it is the tools themselves that can make their own requirements for annotations that need to be “hung” on the specification - this will provide a wider range of possibilities in the development of contractual tools.

The above concept is depicted in the figure below:

image

Among the drawbacks of this approach, one can single out the high complexity of manual synchronization of the base specification file and each of the overlays: when updating the structure of the base specification, the required changes should be applied to the overlay structures. This problem becomes more serious with the appearance of more than one overlay.

A possible and most obvious solution would be to develop a special editor or add-on for the existing online RAML editor https://github.com/mulesoft/api-designer. The editing area remains unchanged, but it becomes possible to create tabs: each new tab is a window for editing the overlay attached to it. When editing the base structure of the specification in the main window, the structures in all created tabs also change, and if an incompatibility of a new structure is detected with already existing annotations placed in tabs overlays, a warning appears. A more detailed consideration of such an editor is a separate topic and deserves serious consideration.

Existing groundwork


In the search for existing solutions that are close to the implementation of the idea of ​​using annotations as a means of describing meta-information, the following solutions were found:

  • https://github.com/raml-org/raml-annotations repository containing official annotations approved by the developer community RAML. In the current version, only OAuth2 annotations are available. Can be used by external tools to obtain meta-information describing aspects of the implementation of OAuth2 for the specification being developed by the API;
  • https://github.com/petrochenko-pavel-a/raml-annotations @ petrochenko-pavel-a library of annotations with logical grouping by scopes. The project is rather experimental, but it perfectly illustrates the idea of ​​using annotations. The most interesting annotation groups are:
    • additionalValidation.raml - annotations for describing additional rules for the validation of specification models. For example, they can be used by the server library to validate requests for the RAML specification;
    • mock.raml - annotations for describing the details of how a mock server is based on the RAML specification;
    • semanticContexts.raml — annotations indicating the semantic context of individual declared structural blocks of the RAML specification;
    • structural.raml - annotations that clarify the role of a separate RAML entity in the overall structure of the described domain model;
    • uiCore.raml is an example of annotations that are possible for use by UI generation tools based on the RAML specification;

The repository also contains libraries of service types suitable for use as primitives when describing data structures of the RAML specification.

RAML problems


Despite the functionality, progressiveness of the basic idea, attention from major software vendors (cisco, spotify, vmware, etc.), today RAML has serious problems that can be fatal regarding its successful fate:

  • Small and isolated open-source community;
  • The incomprehensible strategy of the main developer of RAML is mulesoft . The company develops products that are only a copy of existing OAS-based solutions (included in the Anypoint Platform ), instead of creating services that emphasize the advantages of RAML over Swagger;
  • The consequence of the first paragraph: a small number of open-source libraries / tools;
  • A higher threshold of entry than the OAS (this is strange, but many people think so);
  • Due to the large number of bugs and problems with UX / UI, the main service is completely unusable and repulsive to users . The RAML entry point is https://anypoint.mulesoft.com/ .

Conceptual differences. First conclusion


There are contradictions within the community regarding the basic concept. Someone thinks that RAML is a model definition language , and someone that this is an API definition language as OAS or Blueprint (guys who call themselves RAML developers often mention this in various comments). The concept of a model definition language would allow within the RAML specification to describe the domain model of the domain without being strictly tied to the context of the resource API description, thereby expanding the horizons of options for using the specification with external tools (actually creating a foundation for the existence of a real Single Contract !). This definition of the resource concept can be seen on the readhat docs site (http://restful-api-design.readthedocs.io/en/latest/resources.html , by the way, I recommend everyone to read this wonderful guide on API design):
We call information that We describes available resources types , their behavior, their relationships and the resource model of an API . Resource masturbation mapping of the application data model .
In the RAML, the application data model is the types declared in the types block , and the resource model of an API is what is described in the resources RAML block . Therefore, it is necessary to have the possibility of describing this mapping . But the current implementation of RAML allows you to make such a mapping only 1 to 1, that is, use types “as is” inside the declaration of the resource API.

I consider this the most important problem of the language, the solution of which will allow RAML to go beyond the API definition language and become a full-fledged model definition language: in a more general language (rather than OAS or Blueprint) used to describe single contracts of systems, in essence, which are the formal core of many of their components.

The above makes RAML a weak player who is currently unable to win a Swagger competition. Perhaps that is why, as a result, the main developer of RAML went to drastic measures https://blogs.mulesoft.com/dev/api-dev/open-api-raml-better-together/

Idea Single contract RAML SaaS


Based on the concept of Single Contract , building on the hosting idea of ​​the Swagger hub OAS-based API specifications, and also relying on the capabilities of RAML to declare meta-information and split the base specification using overlays, the idea of ​​an alternative SaaS solution for hosting and managing language-specific RAML Surpass Swagger Hub and Apiary with the volume and quality of the possible functionality.

The new service, by analogy with Swagger hub, will be hosting user contracts with the provision of an online editor and the ability to view documentation-previews with real-time updates. The main difference should be the presence of a catalog of contract plug-ins built into the service, any of which the user can install in his current project an API specification. For installation, it will be required to implement the required RAML annotations specified in the plugin documentation. After adding a new plug-in to the project, in the code editor window a new tab will be added when switching to which, editing annotations of the installed plug-in will become available. The structure of the base specification should be automatically duplicated in all tabs corresponding to the plugins.

image

Technically, each tab will be an abstraction over RAML overlay containing annotations of each specific plug-in. This ensures that the specification is compatible with any tool that supports RAML 1.0.

The plug-in directory must be open to the open source community. It is also possible the implementation of paid plug-ins, which can serve as an incentive to develop new ones.

Possible plugins: API documentation with support for a large number of annotations for flexible parameterization of its rendering, a smart mock server (from the example above), downloadable libraries for validating requests or code generation, debugging tools for outgoing API requests for mobile applications (caching proxy), load tests with test flow setting via annotations, various plugins for integration with external services.

This service idea contains clear advantages over existing services for managing API-specifications and its implementation lays the beginning of a possible change in the approach to the implementation of any external systems one way or another connected with API.

Second conclusion


The purpose of this article is not to criticize Swagger, Apiary, or other de facto standard API development tools, but rather to consider the conceptual difference c by the design approach of specifications promoted by RAML, an attempt to introduce the concept Contract first and consider the possibility of its implementation on the basis of RAML. Another goal was the desire to attract quite deserved attention of developers to RAML for the further possible development of its community.

Official site RAML
Slack channel
Specification

Thank you for your attention.

Also popular now: