How to write an exchange with 50 suppliers and not go crazy
Suppliers are different. Some are ready to adapt to our format - others are not; some exchange SOAP'om - others REST'om; some work with product codes - others with offer identifiers; some are ready to give status on request - others are not; some have directories, the elements of which you need to match with your own - others do not. In general, very different.
Communication with suppliers needs to be automated. 21st century in the yard. It is clear that in an ideal world it is necessary to convene a conference of suppliers and consumers in this industry (in my case, it’s auto parts), at which to agree on a single format for the exchange of information and all will be happy. Pharmacists have this. But we live in an imperfect world.
Everyone has one initial idea: the supplier is ready to give you elements of his directories and is ready to accept an order from you. But all suppliers, business processes have their own characteristics and this imposes different initial requirements on the supplier APIs, and different programmers who implement them bring the picture to the point that the APIs of different suppliers have no common features other than the initial idea.
How to write an exchange in fifty formats? Write fifty exchanges? Let's assume that in the best case, an exchange with one supplier should be written and tested for 1 week and the programmer’s rate is 3000r / hour. Multiplied? And I still did not remember the support and administration of all this zoo.
There are other options? My answer: write one exchange!
We must come up with the 51st supplier and write an exchange with him. It must be the ideal supplier for us, for our processes. This supplier should have common features of all suppliers, differing from them in details.
We must begin to think not “from the supplier”, but “from us” and the picture will instantly simplify. We have one business process and one programmer, too, and everything that goes beyond what is reasonable in our opinion is not particularly interesting to us. It is clear that such an approach will inevitably lead to "coarsening" in logic and some kind of "thin" functionality will go under the knife, but this is the price of simplicity.
Such roughening, I counted three types:
In inventing my ideal supplier, I shoveled the supplier documentation for their API and developed 4 basic methods by which I developed the API of my ideal supplier.
Generalization is the basic principle of developing any kind of functionality. Exchange with suppliers is no exception.
For example, one supplier can use the status of orders “accepted”, “in operation”, “shipped”, “refused”. Another “in the work of the manager”, “in the work at the warehouse”, “in the work of the transport company”, “closed successfully”, “canceled”. I also saw 27 different statuses with another supplier. Seriously, I specifically counted them. It is clear that in our system we do not need this whole zoo. We will summarize the statuses to the minimum set necessary for us: “new”, “in the supplier’s work”, “shipped by the supplier”, “problem”.
Then it remains just to compare each element of the directory of each supplier with an element of our directory.
You can not generalize everything. Sometimes you have to separate.
For example, if one supplier accepts the entire order (all order lines in one XML), and the other according to the principle of the online store basket (add goods to the basket - add goods to the basket - order the basket), then writing this in some “general” way does not will work out.
In this case, the exchange with our ideal 51st supplier should be divided. those. he must optionally be able to send the order in one and the other way. For me this is called the “method of sending an order” and it has two possible meanings: “order entirely” and “line by line”.
And there is also a setting “status check method” and it already has three options: “request by order - return by order”, “request by order - return by line” and “request by line - return by line”.
But to get involved in separation is not worth much, since each fragmentation leads to a complication of the system. And what definitely should not be done is to allow tree-like separation. Try to reduce everything to flat settings. If fragmentation of functionality on one basis will depend on fragmentation on another, you will dig in the settings and instead of elegant simplicity you will get the hell of settings.
The clipping method is intuitive and is applied in cases where the functionality of the provider exceeds the one you need.
For example, a supplier may provide the ability to edit an order up to a specific point (for example, before being transferred to a set).
But this is not particularly interesting for us and all other suppliers do not have such an opportunity, so we just need to say that we will not use this opportunity.
It is necessary to clearly define the boundaries of automation. Anything beyond their scope should be excluded from consideration. Ignore the arguments “this is an additional convenience”: if the rest of the suppliers can be dispensed with, then this can be done.
The default method is the opposite extreme of the clipping method. In this case, the problem is that the automation frontier includes functionality that the provider does not support. For example, returning the status of an order. Some vendors may not support this.
In this case, it is not worthwhile to check in all subsequent algorithms whether the status of the order is considered current or ignore it, because the supplier does not know how to return it. It is much easier to put a “stub”, which will translate such orders into the last successful status.
Thus, by analyzing your needs and capabilities, you can draw the boundaries of automation. This will be the completion of the design phase of the API of your ideal 51st provider. That is, if you go into technology, there will still be 50 suppliers in the system, but the data upload / download code will be the same and will generate XML for all 50 suppliers in the 51st provider format.
It so happened that in this article I focused on unloading the order, as on the most variable and complex part of the exchange (at least for me), but everything written can be applied to loading and unloading both order and master data, and anything else.
Next, you need to consider how best to arrange the upload file for the ideal 51st provider:
Thus, we should get a file containing all the data the supplier needs to accept the order, but in a slightly invalid form)
It's time to move from theory to business, from ideal suppliers to real ones. I think everyone who is interested has already guessed that the XML we got will be converted in a tricky way to get 50 from one format. For this, we will use a two-stage conversion:
First we need to create the data structure itself. XML is best suited for this, because we have a powerful XML document transformation language (XSLT - eXtensible Stylesheet Language Transformations) that allows you to convert an XML document from one view to another.
The language is not complicated, it can be mastered at a basic level in a couple of days, and 99% of my XSLT cost only two instructions value-of and for-each.
Further, the created structure may need to be converted to a format other than XML. In this sense, the XML <-> JSON conversion seems to be the most difficult, I laid out an example of such a conversion. (but I won’t show you, because the UFO forbade the link to the Infostart to be inserted here) There is also a conversion of XML to GET request parameters, i.e. to a line like? item = 12345 & qty = 4 & price = 19.50, but it’s already quite simple. As a result of this step, an exchange text valid for a specific supplier should appear.
After receiving a valid text for exchange - it remains only to send. Functionality which I conditionally call the universal dialer is engaged in this task. She is able to call the settings SOAP or REST service and transfer the result of the converter. The dialer should provide for very different scenarios: it should be ready to send data as parameters and as the request body, POST and GET methods, with basic and not very authorization, etc.
Theoretically, you can tie at least read / write Excel to the Dialer, select specific “connectors” to fit your needs.
In principle, the functions of obtaining data in the format of the ideal 51st provider, XSL transformation, format converter and dialer can be moved to a separate instance without linking this functionality to any of the existing systems, which I did. I call this thing Interface Converter.
Interface Converter code fits in 300 lines. Basic settings (XSLT texts, transformation and call settings) are stored as data. As a result, the process of sending an order to the supplier is as follows:
All this happens "on the fly", i.e. synchronously, i.e. ERP thinks that it is exchanging with all suppliers in one format convenient to it, without even noticing that something is wrong. This approach, of course, is applicable only for the transmission of relatively small data packets. If you have 1 GB price lists, you should consider how to fasten the asynchronous to this whole story.
Almost every exchange has values that are simply hardcoded. In part, this may be due to the too rich functionality of the supplier, for example, the parameter “percentage by which the price of the goods can be increased without additional coordination”, in part, these are just some values like “for you it is always 4000”. What is 4000? Why 4000? Just 4000 and that's it. In order not to store such “dead” values in ERP, the Converter has the ability to enrich the ERP data with some other XML. Those. it is just arbitrary XML that is stored in the interface converter database.
Each interface (provider) has its own XML. Additional enrichment takes place according to the principle of simple gluing with a wrapper in a common tag, for example:
And further, such enriched XML is fed to the input of the XSLT engine, where both the data sent by ERP and our static data are available.
We didn’t just start to fence this whole garden for no reason, but because we wanted to reduce the time for connecting a supplier. Here's what a new provider connection is like:
In practice, all these actions take up to 3 hours. It is clear that there are times when the supplier thinks up something "kind of", but in 95% of cases you can meet within 2-3 hours. (I made a mistake in this sentence and wrote “substitute” instead of “supplier.” Coincidence? ..)
Naturally, there are still a lot of details and nuances that it is not possible to talk about in the review article, but the purpose of the article is to show one of the possible approaches to the design of multi-format exchange architecture, I consider it achieved.
Communication with suppliers needs to be automated. 21st century in the yard. It is clear that in an ideal world it is necessary to convene a conference of suppliers and consumers in this industry (in my case, it’s auto parts), at which to agree on a single format for the exchange of information and all will be happy. Pharmacists have this. But we live in an imperfect world.
Everyone has one initial idea: the supplier is ready to give you elements of his directories and is ready to accept an order from you. But all suppliers, business processes have their own characteristics and this imposes different initial requirements on the supplier APIs, and different programmers who implement them bring the picture to the point that the APIs of different suppliers have no common features other than the initial idea.
How to write an exchange in fifty formats? Write fifty exchanges? Let's assume that in the best case, an exchange with one supplier should be written and tested for 1 week and the programmer’s rate is 3000r / hour. Multiplied? And I still did not remember the support and administration of all this zoo.
There are other options? My answer: write one exchange!
We must come up with the 51st supplier and write an exchange with him. It must be the ideal supplier for us, for our processes. This supplier should have common features of all suppliers, differing from them in details.
We must begin to think not “from the supplier”, but “from us” and the picture will instantly simplify. We have one business process and one programmer, too, and everything that goes beyond what is reasonable in our opinion is not particularly interesting to us. It is clear that such an approach will inevitably lead to "coarsening" in logic and some kind of "thin" functionality will go under the knife, but this is the price of simplicity.
Such roughening, I counted three types:
- Functionality that we need, but which some suppliers will not be ready to provide
- The functionality that suppliers provide, but we do not need it.
- Functionality implemented by radically different methods from different suppliers.
In inventing my ideal supplier, I shoveled the supplier documentation for their API and developed 4 basic methods by which I developed the API of my ideal supplier.
Method 1: Generalize
Generalization is the basic principle of developing any kind of functionality. Exchange with suppliers is no exception.
For example, one supplier can use the status of orders “accepted”, “in operation”, “shipped”, “refused”. Another “in the work of the manager”, “in the work at the warehouse”, “in the work of the transport company”, “closed successfully”, “canceled”. I also saw 27 different statuses with another supplier. Seriously, I specifically counted them. It is clear that in our system we do not need this whole zoo. We will summarize the statuses to the minimum set necessary for us: “new”, “in the supplier’s work”, “shipped by the supplier”, “problem”.
Then it remains just to compare each element of the directory of each supplier with an element of our directory.
Method 2: Separation
You can not generalize everything. Sometimes you have to separate.
For example, if one supplier accepts the entire order (all order lines in one XML), and the other according to the principle of the online store basket (add goods to the basket - add goods to the basket - order the basket), then writing this in some “general” way does not will work out.
In this case, the exchange with our ideal 51st supplier should be divided. those. he must optionally be able to send the order in one and the other way. For me this is called the “method of sending an order” and it has two possible meanings: “order entirely” and “line by line”.
And there is also a setting “status check method” and it already has three options: “request by order - return by order”, “request by order - return by line” and “request by line - return by line”.
But to get involved in separation is not worth much, since each fragmentation leads to a complication of the system. And what definitely should not be done is to allow tree-like separation. Try to reduce everything to flat settings. If fragmentation of functionality on one basis will depend on fragmentation on another, you will dig in the settings and instead of elegant simplicity you will get the hell of settings.
Method 3: Clipping
The clipping method is intuitive and is applied in cases where the functionality of the provider exceeds the one you need.
For example, a supplier may provide the ability to edit an order up to a specific point (for example, before being transferred to a set).
But this is not particularly interesting for us and all other suppliers do not have such an opportunity, so we just need to say that we will not use this opportunity.
It is necessary to clearly define the boundaries of automation. Anything beyond their scope should be excluded from consideration. Ignore the arguments “this is an additional convenience”: if the rest of the suppliers can be dispensed with, then this can be done.
Method 4: Default
The default method is the opposite extreme of the clipping method. In this case, the problem is that the automation frontier includes functionality that the provider does not support. For example, returning the status of an order. Some vendors may not support this.
In this case, it is not worthwhile to check in all subsequent algorithms whether the status of the order is considered current or ignore it, because the supplier does not know how to return it. It is much easier to put a “stub”, which will translate such orders into the last successful status.
Thus, by analyzing your needs and capabilities, you can draw the boundaries of automation. This will be the completion of the design phase of the API of your ideal 51st provider. That is, if you go into technology, there will still be 50 suppliers in the system, but the data upload / download code will be the same and will generate XML for all 50 suppliers in the 51st provider format.
It so happened that in this article I focused on unloading the order, as on the most variable and complex part of the exchange (at least for me), but everything written can be applied to loading and unloading both order and master data, and anything else.
Next, you need to consider how best to arrange the upload file for the ideal 51st provider:
- First, it must be XML (why - it will become clear later).
- Secondly, it should contain the maximum amount of data that you can only upload with this order. Those. at this stage, it doesn’t matter to us whether it is required to indicate the VAT rate for the supplier to whom the order is ultimately intended — we must unload everything that may be required.
- Thirdly, convertible values must be converted. Those. already at the stage of XML formation for the ideal 51st supplier, for example, product identifiers should be substituted from the supplier to whom the order is intended. This should not cause any difficulties, since all the data for this is in the order.
Thus, we should get a file containing all the data the supplier needs to accept the order, but in a slightly invalid form)
So how does it work?
It's time to move from theory to business, from ideal suppliers to real ones. I think everyone who is interested has already guessed that the XML we got will be converted in a tricky way to get 50 from one format. For this, we will use a two-stage conversion:
- XSLT
- Format converter
First we need to create the data structure itself. XML is best suited for this, because we have a powerful XML document transformation language (XSLT - eXtensible Stylesheet Language Transformations) that allows you to convert an XML document from one view to another.
The language is not complicated, it can be mastered at a basic level in a couple of days, and 99% of my XSLT cost only two instructions value-of and for-each.
Further, the created structure may need to be converted to a format other than XML. In this sense, the XML <-> JSON conversion seems to be the most difficult, I laid out an example of such a conversion. (but I won’t show you, because the UFO forbade the link to the Infostart to be inserted here) There is also a conversion of XML to GET request parameters, i.e. to a line like? item = 12345 & qty = 4 & price = 19.50, but it’s already quite simple. As a result of this step, an exchange text valid for a specific supplier should appear.
After receiving a valid text for exchange - it remains only to send. Functionality which I conditionally call the universal dialer is engaged in this task. She is able to call the settings SOAP or REST service and transfer the result of the converter. The dialer should provide for very different scenarios: it should be ready to send data as parameters and as the request body, POST and GET methods, with basic and not very authorization, etc.
Theoretically, you can tie at least read / write Excel to the Dialer, select specific “connectors” to fit your needs.
In principle, the functions of obtaining data in the format of the ideal 51st provider, XSL transformation, format converter and dialer can be moved to a separate instance without linking this functionality to any of the existing systems, which I did. I call this thing Interface Converter.
Interface Converter code fits in 300 lines. Basic settings (XSLT texts, transformation and call settings) are stored as data. As a result, the process of sending an order to the supplier is as follows:
- ERP generates an order and sends it in the format of the 51st supplier to Interface Converter
- The converter receives the XML test in the format of the 51st provider and enriches it
- determines which provider to send the request to (conditionally, by the interface code), - on the fly performs XSL transformation and format conversion
- immediately sends the converted request further to the supplier
- receives a response from the supplier
- from the response body in the reverse order, performs format conversion, enrichment, then XSL transformation into the response format of the ideal provider
- returns the ideal provider response to ERP
All this happens "on the fly", i.e. synchronously, i.e. ERP thinks that it is exchanging with all suppliers in one format convenient to it, without even noticing that something is wrong. This approach, of course, is applicable only for the transmission of relatively small data packets. If you have 1 GB price lists, you should consider how to fasten the asynchronous to this whole story.
Enrichment chip
Almost every exchange has values that are simply hardcoded. In part, this may be due to the too rich functionality of the supplier, for example, the parameter “percentage by which the price of the goods can be increased without additional coordination”, in part, these are just some values like “for you it is always 4000”. What is 4000? Why 4000? Just 4000 and that's it. In order not to store such “dead” values in ERP, the Converter has the ability to enrich the ERP data with some other XML. Those. it is just arbitrary XML that is stored in the interface converter database.
Each interface (provider) has its own XML. Additional enrichment takes place according to the principle of simple gluing with a wrapper in a common tag, for example:
...данные из ERP...
...статичные данные Конвертера...
And further, such enriched XML is fed to the input of the XSLT engine, where both the data sent by ERP and our static data are available.
What do we have as a result?
We didn’t just start to fence this whole garden for no reason, but because we wanted to reduce the time for connecting a supplier. Here's what a new provider connection is like:
- Exploring API Documentation
- Creation of an appropriate provider in ERP and its configuration (about 10 checkmarks).
- XSLT Exchange Text Conversion Description
- Description of format conversion (if required)
- Outgoing connection settings (10 more checkmarks).
In practice, all these actions take up to 3 hours. It is clear that there are times when the supplier thinks up something "kind of", but in 95% of cases you can meet within 2-3 hours. (I made a mistake in this sentence and wrote “substitute” instead of “supplier.” Coincidence? ..)
Naturally, there are still a lot of details and nuances that it is not possible to talk about in the review article, but the purpose of the article is to show one of the possible approaches to the design of multi-format exchange architecture, I consider it achieved.