June 9, 2011 at 15:20Integration of Web-applications with commodity accounting systems - text of the report from the conference DUMP-IT.RU
On May 30, we made a number of reports in different sections of the DUMP conference , which was held in Yekaterinburg. Initially, we were asked to touch upon the holistic topic “why 1C-Bitrix”, shit :-), but we decided to confine ourselves to the problems of customizing boxes when creating sites synchronized with commodity accounting systems (hereinafter referred to as STU).
So, here was the report.
Artsofte most often faces the challenges of integrating an online store with 1C \ Scala \ SuperMAG and online payment systems. We also introduce, but are not limited to, systems for tracking the status of an order. The pool of integrated solutions may also include more complex systems, such as customer-side online banking, integration with billing systems, etc.
We are engaged exclusively in custom development on Symfony (PHP) and .NET, and often we are contacted by clients who already have all sorts of experience.
But what if, as one of the links, an alternative solution is used: CMS or a custom system on the one hand and alternative software for commodity accounting on the other hand?
In this case, things are not so rosy. If the commodity accounting system or CMS does not have synchronization tools, or these tools are incompatible, it is necessary to refine one or both links, which can turn out to be significant costs for the customer both in time and in cost. These costs may even exceed the cost of the rest of the work, or force the customer to switch to another solution (meaning a solution for goods accounting or a store)
One way or another, “complete” or taken directly “out of the box”, such a solution will eventually exhaust its limits of standard functionality. The box will be full. Dead end.
The scenarios described above, in addition to integration problems per se, can have “side effects” for the customer.
Our seo-division was engaged in the promotion of the site detsad.ru, developed by another company. This site is a simple online store with a three-level directory hierarchy and configured synchronization with 1C-Accounting, by manually downloading the CSV file through the CMS.
The path to the catalog page (in this case, a link with the maximum level of nesting is provided) looks like this: www.detsad.ru/catalogue/200/3598 , where the last two numbers are the category and product number, respectively.
The problem was that these numbers correspond to identifiers in 1C, which had the property to change. As a result of synchronization, old links to products became incorrect and, as a result, the site lost ground in search engines. As a result, we rewrote the parser.
Here is another example. Let's say my friends and I decided to have a beer. We love Hugarden, and decided to order a box. We love to buy at the Zvezdny supermarket, but we are too lazy to go to it. Therefore, we decided to order beer at the Zvezdny online store. Once on the main page of the online store, we select: “Beer”, “San Inbev”, “SB”, ... what the hell is that? What else is “San Inbev” ?! What kind of "SB", "reinforced concrete", "PB"? Reinforced concrete or what?
I directly imagine myself going into the store and looking at the huge sign: “Beer”. I wrap it up in the “Sun Inbev” section, look for the “SB” rack, and on it ... Oh well, no, already I got a little sick. I’ll go better to buy drinking water. To her is not so deep.
As a result, the client understands that he needs a new solution that will not only fully take into account the business logic, but also improve along with his business.
Based on our experience, we can assume with a high degree of probability that during the integration of the STU and the Web frontend, the developer simply copies the directory structure from the STU.
The result is low conversion.
When we start a custom integration project, in most cases we have to face the fact that companies, as a rule, do not have a developed and implemented IT strategy. The rules and policies of the development, implementation, use of information systems, principles and methods of integration are not described. Starting work on a project, you have to develop these documents from scratch, formalize business processes. Work on such projects usually begins with a formalization of the task we call the “Black Box Model”.
The diagram of this model is as follows:
At the entrance, there is usually a development task that contains the requirements of the customer in general form, as well as general data on the used commodity accounting systems (STU). The developed Web application and STU are black boxes, the internal structure of which, if known, is usually in the most general terms. The first stage of work on the project is the decomposition of the Web application, which allows you to understand the requirements for its own architecture, and based on these requirements build the logic of interaction with the STU.
But why not immediately start the implementation with a detailed analysis and not immediately come to a custom solution?
In most cases, the answer is simple. Offering their services to a customer, companies evaluate the necessary amount of work differently,and when offers differ in price up to 30 times (the real case from our practice with the B2B store shop.nag.ru ), offers from the lower price segment look most attractive to the customer, but at the same time they contain a catch.
Being one of the key factors, the low cost of implementing the system does not mean at all that we have saved, and when choosing a solution, we need to evaluate in a broader perspective, for example, calculate the cost of ownership of an integrated solution, and it will include:
The last item often becomes the basis of the project budget.
The lack of mention of the cost of maintenance in the commercial proposal creates a rosy picture for the customer. The thing is that the boxed solution will be really inexpensive in terms of implementation.
An online store on Bitrix or VirtueMart with a template design is deployed in a couple of hours, not counting the content. After that, unloading is just as quick. But what if the necessary information block is not in the system or the description of the data exchange format does not contain information about the stock balances? "Finish" large and, especially, commercial CMS is very difficult. And therefore it is very expensive.
So how can you "make friends" STU with a Web application? To answer this question, we first consider possible integration models.
Let us turn to the experience of our Western colleagues. It is worth noting that in the Western market the number of different products and solutions is much larger than ours, and the opportunities offered for interaction are often very fragmented. In addition, many systems require much more complex and deeper integration than unloading product items from an ERP system. To ensure flexible and reliable integration in these cases, the so-called Middleware, an intermediate layer that provides data exchange between disparate subsystems, such as eCommerce, ERP, Web services, etc. Combining many formats and data transfer protocols, providing the ability to convert one format to another, middleware also provides a high degree of reliability and flexibility of the integrated system, since a change in one of its components will entail only a change in the rules of interaction with it. Sometimes, speaking of integration Middleware, the concept of an integration bus is introduced, which provides a single data exchange channel for all nodes of the system, regardless of their origin and territorial location.
A typical interaction scheme looks like the one shown below:
Architecture of the interaction of Pervasive Data Integrator Middleware with applications.
The arrows indicate the direction of data transfer and the protocols used for various tasks. We list the most popular of them.
Data transfer via these protocols is usually provided by HTTP / S and FTP / S transports.
Having the necessary tools, it often turns out to be sufficient to describe the data structures and choose the most suitable exchange protocol.
But in view of the still low level of automation of business processes, the task is narrowing on the one hand, and on the other hand becoming more complicated. Firstly, the market for web integration in Russia, according to experts, is underdeveloped, which means that there is simply no necessary Middleware. Western products are also often not suitable for us in connection with the tradition of development in Russia, when each company seeks to have its own solution. But on the other hand, a small number of vendors narrows the set of possible cases and often reduces the task of integration to the need to integrate a goods accounting system with an online storefront or online store. We divided the interactions of the Web application and the product system, depending on the complexity of the solution, into 3 levels in the areas of data exchange and data sets transmitted in both directions.
Read more about them here.
To ensure interactions at all three levels, we use the so-called. interface layer. This layer allows you to upload the data necessary for synchronization in the required format that meets the requirements of business logic. At the same time, we do not use bloated XML standards like CommerceML, but we adapt the protocols to the needs of a particular model.
When designing interactions between web applications and STUs, we usually work closely with specialists in this STU.
This allows us to receive data in a format convenient for processing and not to limit ourselves to the built-in export capabilities of STU data. An extension (upload) is connected to the STU, which provides data for processing on the side of the Web application.
In the case of two-way exchange, this extension also assumes the functions of transferring data to the STU. The web application contains the same module that parses and updates the database, as well as triggers synchronization-related events.
In the case of two-way communication, the Web application interface module is responsible for the aggregation and serialization of exported data. Ok, now STU and Web application can understand each other. However, interactions need to be somehow initiated. We immediately note the option of manual initiation. Since our task is to automate the interaction process as much as possible, we consider only automatic methods of initiation.
The manual method remains as a fallback in emergency cases when one of the services fails. Automatic initiation of synchronization implies that under certain well-defined conditions, one of the sides of the interface layer raises a synchronization event, and the second side processes this event.
Today we use two different ways to initiate synchronization, depending on the available data transmission channels and the technical capabilities of the STU.
Uploading to an XML file implies that the data is saved as an XML file in a specific location on the FTP server. The web application periodically checks the files stored on the server for updates and, if a new version is detected, synchronizes. The interface extension of STU behaves in a similar way.
In the initiator-listener model, the Web application acts as a listener, and the STU interface module sends requests to the Web application, as a result of which actions to send or receive information are initiated in the Web application. This model does not require any third-party server, since the Web application performs its functions, but it requires the Web client functions in the STU interface module. Such functions, for example, are in 1C v8.
As for the analysis of data and updating the database of the Web application, for each application such a module is designed separately taking into account the current and predicted needs of business logic, as well as performance and fault tolerance requirements. The range of solutions ranges from simple XML parsing and subsequent sequential entry of data into tables to multi-threaded transactional models with integrity control and model-level triggers. In this case, by model I mean the level of MVC architecture used in studio projects.
As can be seen from the above, customization of the integration solution for the customer’s business processes begins already at the stage of designing interactions with the STU. However, this is only a small part. More urgently, the need for a custom integracin solution begins to be felt when it comes to such things as:
When developing a custom product, we are practically unlimited in our decisions, since we do not impose on ourselves either an architectural framework or an instrumental one. We have our own component base, and we can do anything we want.
Well, that’s it! =)
So, here was the report.
Artsofte most often faces the challenges of integrating an online store with 1C \ Scala \ SuperMAG and online payment systems. We also introduce, but are not limited to, systems for tracking the status of an order. The pool of integrated solutions may also include more complex systems, such as customer-side online banking, integration with billing systems, etc.
Problem
We are engaged exclusively in custom development on Symfony (PHP) and .NET, and often we are contacted by clients who already have all sorts of experience.
When approaching customers, we often come across 3 options for hopelessness.
- a) The client does not have a clearly formulated task and asks the question: “What if you try?” In this case, we recommend that the client first play with one of the boxed versions, and there see p.
- b) The client decides to create his own development, turns to the decision of the lower price segment, and ultimately receives a “bicycle”. And, it would seem, all is well, the customer is satisfied. He periodically uploads goods to the site, clogs orders in 1C. But after a while it turns out that the wheels of the bicycle are square, the seat is not adjustable, and the speed switch is not provided in principle. Without the support of the creator of his system, the client finds himself in a deadlock.
- c) The client chose a “boxed” solution. It so happened historically that despite the presence of various vendors of STU solutions, such as Oracle, Scala, SuperMag, etc., in the vast majority of cases, STU is understood as a 1C: Enterprise complex in all its configurations. Therefore, when it comes to the integration of an online store with a commodity accounting system or an accounting system, the first things to consider are solutions that provide integration with 1C out of the box. And it is logical that it comes primarily to a bunch of 1C-Enterprise and 1C-Bitrix. This solution is really quite simple to implement, and for some time things are much better than in the first version. The web application will be developed along with the requirements of the customer’s business.
But what if, as one of the links, an alternative solution is used: CMS or a custom system on the one hand and alternative software for commodity accounting on the other hand?
In this case, things are not so rosy. If the commodity accounting system or CMS does not have synchronization tools, or these tools are incompatible, it is necessary to refine one or both links, which can turn out to be significant costs for the customer both in time and in cost. These costs may even exceed the cost of the rest of the work, or force the customer to switch to another solution (meaning a solution for goods accounting or a store)
One way or another, “complete” or taken directly “out of the box”, such a solution will eventually exhaust its limits of standard functionality. The box will be full. Dead end.
The scenarios described above, in addition to integration problems per se, can have “side effects” for the customer.
An example of problems caused by the integration curve:
1. Case "detsad.ru"
Our seo-division was engaged in the promotion of the site detsad.ru, developed by another company. This site is a simple online store with a three-level directory hierarchy and configured synchronization with 1C-Accounting, by manually downloading the CSV file through the CMS.
The path to the catalog page (in this case, a link with the maximum level of nesting is provided) looks like this: www.detsad.ru/catalogue/200/3598 , where the last two numbers are the category and product number, respectively.
The problem was that these numbers correspond to identifiers in 1C, which had the property to change. As a result of synchronization, old links to products became incorrect and, as a result, the site lost ground in search engines. As a result, we rewrote the parser.
2. Case "Star"
Here is another example. Let's say my friends and I decided to have a beer. We love Hugarden, and decided to order a box. We love to buy at the Zvezdny supermarket, but we are too lazy to go to it. Therefore, we decided to order beer at the Zvezdny online store. Once on the main page of the online store, we select: “Beer”, “San Inbev”, “SB”, ... what the hell is that? What else is “San Inbev” ?! What kind of "SB", "reinforced concrete", "PB"? Reinforced concrete or what?
I directly imagine myself going into the store and looking at the huge sign: “Beer”. I wrap it up in the “Sun Inbev” section, look for the “SB” rack, and on it ... Oh well, no, already I got a little sick. I’ll go better to buy drinking water. To her is not so deep.
As a result, the client understands that he needs a new solution that will not only fully take into account the business logic, but also improve along with his business.
Based on our experience, we can assume with a high degree of probability that during the integration of the STU and the Web frontend, the developer simply copies the directory structure from the STU.
The result is low conversion.
When we start a custom integration project, in most cases we have to face the fact that companies, as a rule, do not have a developed and implemented IT strategy. The rules and policies of the development, implementation, use of information systems, principles and methods of integration are not described. Starting work on a project, you have to develop these documents from scratch, formalize business processes. Work on such projects usually begins with a formalization of the task we call the “Black Box Model”.
The diagram of this model is as follows:
At the entrance, there is usually a development task that contains the requirements of the customer in general form, as well as general data on the used commodity accounting systems (STU). The developed Web application and STU are black boxes, the internal structure of which, if known, is usually in the most general terms. The first stage of work on the project is the decomposition of the Web application, which allows you to understand the requirements for its own architecture, and based on these requirements build the logic of interaction with the STU.
But why not immediately start the implementation with a detailed analysis and not immediately come to a custom solution?
In most cases, the answer is simple. Offering their services to a customer, companies evaluate the necessary amount of work differently,and when offers differ in price up to 30 times (the real case from our practice with the B2B store shop.nag.ru ), offers from the lower price segment look most attractive to the customer, but at the same time they contain a catch.
Being one of the key factors, the low cost of implementing the system does not mean at all that we have saved, and when choosing a solution, we need to evaluate in a broader perspective, for example, calculate the cost of ownership of an integrated solution, and it will include:
- Actually, the cost of the main software components
- The cost of implementing software components, if required
- The cost of implementing an integration solution
- Costs for subsequent customization of the integration solution
The last item often becomes the basis of the project budget.
The lack of mention of the cost of maintenance in the commercial proposal creates a rosy picture for the customer. The thing is that the boxed solution will be really inexpensive in terms of implementation.
An online store on Bitrix or VirtueMart with a template design is deployed in a couple of hours, not counting the content. After that, unloading is just as quick. But what if the necessary information block is not in the system or the description of the data exchange format does not contain information about the stock balances? "Finish" large and, especially, commercial CMS is very difficult. And therefore it is very expensive.
Integration Models:
So how can you "make friends" STU with a Web application? To answer this question, we first consider possible integration models.
Western model
Let us turn to the experience of our Western colleagues. It is worth noting that in the Western market the number of different products and solutions is much larger than ours, and the opportunities offered for interaction are often very fragmented. In addition, many systems require much more complex and deeper integration than unloading product items from an ERP system. To ensure flexible and reliable integration in these cases, the so-called Middleware, an intermediate layer that provides data exchange between disparate subsystems, such as eCommerce, ERP, Web services, etc. Combining many formats and data transfer protocols, providing the ability to convert one format to another, middleware also provides a high degree of reliability and flexibility of the integrated system, since a change in one of its components will entail only a change in the rules of interaction with it. Sometimes, speaking of integration Middleware, the concept of an integration bus is introduced, which provides a single data exchange channel for all nodes of the system, regardless of their origin and territorial location.
A typical interaction scheme looks like the one shown below:
Architecture of the interaction of Pervasive Data Integrator Middleware with applications.
The arrows indicate the direction of data transfer and the protocols used for various tasks. We list the most popular of them.
Web Services Technologies and Protocols
- Simple Object Access Protocol (SOAP) - is an extension of XML-RPC and provides the ability to exchange arbitrary messages in XML format. Initially used to implement remote procedure call (RPC). Allows the Web service to provide reusable components. Used on top of text protocols, mostly HTML
- Representational State Transfer (REST) - in this case, data is transmitted using a small number of standard formats - HTML, XML, JSON. The network transfer protocol must support caching and must not store state information. Data exchange is built in the form of a request-response. In some cases, it can be used to transfer SOAP objects.
- XML is typically used to convey messages and objects. It has a hierarchical structure and great flexibility.
- Message Queuing - There is a message pool, they are stored until processing.
- CommerceML is the Russian standard for synchronization with 1C: Bookkeeping, actively promoted by 1C and other mastodons, is absolutely stupid because it cannot be international and, as a result, is not supported by large ERPs, however, it deserves attention due to the native implementation in 1C v8
Data transfer via these protocols is usually provided by HTTP / S and FTP / S transports.
Having the necessary tools, it often turns out to be sufficient to describe the data structures and choose the most suitable exchange protocol.
Web and commodity accounting in Russia
But in view of the still low level of automation of business processes, the task is narrowing on the one hand, and on the other hand becoming more complicated. Firstly, the market for web integration in Russia, according to experts, is underdeveloped, which means that there is simply no necessary Middleware. Western products are also often not suitable for us in connection with the tradition of development in Russia, when each company seeks to have its own solution. But on the other hand, a small number of vendors narrows the set of possible cases and often reduces the task of integration to the need to integrate a goods accounting system with an online storefront or online store. We divided the interactions of the Web application and the product system, depending on the complexity of the solution, into 3 levels in the areas of data exchange and data sets transmitted in both directions.
- One-way communication of the nomenclature from 1C and its conclusion to the web-window of the site
- Two-way communication of nomenclature and counterparties from 1C with a web-showcase of a site and a personal account
- Two-way communication of nomenclature, counterparties and orders from 1C with a web-showcase of a site and a personal account
Read more about them here.
To ensure interactions at all three levels, we use the so-called. interface layer. This layer allows you to upload the data necessary for synchronization in the required format that meets the requirements of business logic. At the same time, we do not use bloated XML standards like CommerceML, but we adapt the protocols to the needs of a particular model.
When designing interactions between web applications and STUs, we usually work closely with specialists in this STU.
This allows us to receive data in a format convenient for processing and not to limit ourselves to the built-in export capabilities of STU data. An extension (upload) is connected to the STU, which provides data for processing on the side of the Web application.
In the case of two-way exchange, this extension also assumes the functions of transferring data to the STU. The web application contains the same module that parses and updates the database, as well as triggers synchronization-related events.
In the case of two-way communication, the Web application interface module is responsible for the aggregation and serialization of exported data. Ok, now STU and Web application can understand each other. However, interactions need to be somehow initiated. We immediately note the option of manual initiation. Since our task is to automate the interaction process as much as possible, we consider only automatic methods of initiation.
The manual method remains as a fallback in emergency cases when one of the services fails. Automatic initiation of synchronization implies that under certain well-defined conditions, one of the sides of the interface layer raises a synchronization event, and the second side processes this event.
Today we use two different ways to initiate synchronization, depending on the available data transmission channels and the technical capabilities of the STU.
- Upload to an XML file and transfer it via FTP
- Initiator-Listener Model
Uploading to an XML file implies that the data is saved as an XML file in a specific location on the FTP server. The web application periodically checks the files stored on the server for updates and, if a new version is detected, synchronizes. The interface extension of STU behaves in a similar way.
In the initiator-listener model, the Web application acts as a listener, and the STU interface module sends requests to the Web application, as a result of which actions to send or receive information are initiated in the Web application. This model does not require any third-party server, since the Web application performs its functions, but it requires the Web client functions in the STU interface module. Such functions, for example, are in 1C v8.
As for the analysis of data and updating the database of the Web application, for each application such a module is designed separately taking into account the current and predicted needs of business logic, as well as performance and fault tolerance requirements. The range of solutions ranges from simple XML parsing and subsequent sequential entry of data into tables to multi-threaded transactional models with integrity control and model-level triggers. In this case, by model I mean the level of MVC architecture used in studio projects.
Customization
As can be seen from the above, customization of the integration solution for the customer’s business processes begins already at the stage of designing interactions with the STU. However, this is only a small part. More urgently, the need for a custom integracin solution begins to be felt when it comes to such things as:
- Interactivity and close connections between the components of the user interface in both the server and client parts. This is especially true for tasks performed on the client side and communicating with the Web application server via AJAX requests. Such interactions require a large number of different representations of the data, which are difficult to form on the basis of boxed solutions.
- Data views, flexible filters, highly customizable formats and search results
- Opportunities for order management on the part of not only the client, but also the call center operator. Moreover, in the latter case, the specifics of work require that the functionality of the operator’s user interfaces be configured for maximum productivity.
- Possibilities of introducing a simple hierarchy and “pseudo-categories”. By pseudo-categories we mean entities that functionally correspond to one of the sections of the catalog, but do not have a comparison with the essence of this level in the database. Pseudo-categories include, for example, “our products” or “discounted products”
- And much more.
When developing a custom product, we are practically unlimited in our decisions, since we do not impose on ourselves either an architectural framework or an instrumental one. We have our own component base, and we can do anything we want.
Well, that’s it! =)