June 6, 2014 at 15:07Transcoder - another look at the NSI
Over the years, the company usually comes to the fact that many directories on the same topic (for example, "Hobbies" or "Gender") are stored in completely unimaginable formats in various systems, preventing effective integration and data exchange.
The traditional approach to solving this problem recommends creating a single version of the "Hobby" directory and setting up the exchange flows into it (and from it) for all information systems. We decided to go the other way and create a decentralized open-source NSI - Recoder.
Want to know what we did, and what role do Lucene and Apache CXF play in our product?
How is the storage of directories usually solved?
The IT director decides to stop the species diversity of directories, and the company begins to build a single directory. It will be called upon to combine all possible types of corporate data and finally put an end to “patchwork automation” and “historical heritage”.
If we solve the problem, then immediately for everything, right?
But no.
They will not write about the terrible minuses of centralization in the presentations of large Western companies, who really want to sell you multi-million dollar software, and will not say in press releases. But starting such a project, you will get stuck in a long tedious implementation that will last for years. Because the entities that you will bring to one standard have a different nature. And this makes it unreasonable to combine them within a single system.
Do not believe it - look at the table with data types that are usually found in the company:
In practice, the fastest and most successful implementations take place in those companies that separately solve the goods and materials problem, separately - the task of constructing a directory of counterparties, and somehow deal with classifiers along the way.
Product Information Management (PIM) is usually implemented for goods and materials, Customer Data Integration (CDI) for counterparts, but the interesting thing happens with classifiers: for them, NMI (MDM) is often implemented separately and a single version of the truth is formed.
Is it right to implement NSI and create a single version of classifiers? Or can you solve this problem differently?
Is it good to have a single version of the truth for classifiers?
When classifiers are brought to a single form, one of the scenarios usually occurs:
In general, a single version of the truth is usually expensive.
In addition, with a single standard, the problem of a “gray zone” always arises. This is the difference between the values in the source systems and the values of the coveted unified version of the truth, which were personally confirmed by a specially appointed expert. The presence of a “gray zone” and the need to work with it leads to the appearance of numerous workflows and alternative flows, which is a significant part of the functionality (read the cost) of the NSI.
And also the business is not always ready to wait for numerous approvals for years. Therefore, in the end, the data begins to be forwarded "under the table", using "temporary" transcodes (what it is, we will describe below).
A logical question arises: if a single version of the truth is so expensive, can you do without it?
You can - there are organizations in which NSIs are not implemented. Having carefully looked, you will find in these organizations many small correspondence tables between the reference books “Sex”, “Education”, etc. in different systems. They will be stored either in Oracle, then on the bus, or even directly in the code for download-upload and export-import. Because correspondence tables are the most natural way to solve a problem when the same domain is designated differently for different people or systems.
In other words, you can force all people on earth to speak Esperanto (create a single version of the truth), or you can invite a Chinese-German medical translator to negotiate with Chinese and German doctors (make a correspondence table).
Therefore, we decided to make a simultaneous translator that understands the language of any system and allows them to agree on the same concepts within the framework of the business process. We called the correspondence tables the transcodes, and the system itself - the Transcoder.
The main purpose of the Transcoder is to store copies of directories of the source systems and transcoding between them.
What can I do with the Transcoder
It is possible and necessary:
Classifier "Customer Service Program", in which the attributes are for some reason called non-Russian words Description and Full Description
The system is divided into modules to isolate the API, business logic and storage from each other. The Spring Framework acts as the connecting IoC container between the components.
The technological scheme of data flows can be represented as follows:
As you can see, services do not interact directly with the database at all. Instead, all the work (read / write) goes with the Lucene index, and the results of the write operations are additionally written to the database. Why is this done? Everything is simple: the main task of the system is to quickly find the transcoding (correspondence between entries in a pair of directories), and such an architecture most effectively copes with this on large amounts of data. Plus, we get a full-text search for all the data “out of the box”. Of course, one could use higher-level solutions, such as Elasticsearch or Solr, but they have their own drawbacks, which are more expensive to fight than writing your own code.
In order to ensure the integrity of information, data modification is performed in one transaction in the search index and in the database. Moreover, the interfaces are designed so that the business logic of the application knows only about the index, and the DBMS can be replaced with another solution with minimal modifications. In fact, you need to implement a couple of classes and add them to the classpath - Spring and the class compilation factories on the fly (runtime code compiling) will do the rest.
The framework for SOAP and RESTfull interfaces is Apache CXF, which makes it easy to configure the version of the API (if you suddenly need to change it) and wedge into any phase of the processing of incoming / outgoing messages.
github.com/hflabs/perecoder
rcd.hflabs.ru/rcd/admin/login
Login: admin
Password: demo
confluence.hflabs.ru/x/RICSCg
We started the implementation of the transcoder with several customers.
We welcome your comments on functionality: what you like, what not, what is incomprehensible.
Well, if this product is useful to you, it means that we did not waste our time writing this article;)
The traditional approach to solving this problem recommends creating a single version of the "Hobby" directory and setting up the exchange flows into it (and from it) for all information systems. We decided to go the other way and create a decentralized open-source NSI - Recoder.
Want to know what we did, and what role do Lucene and Apache CXF play in our product?
How is the storage of directories usually solved?
The IT director decides to stop the species diversity of directories, and the company begins to build a single directory. It will be called upon to combine all possible types of corporate data and finally put an end to “patchwork automation” and “historical heritage”.
If we solve the problem, then immediately for everything, right?
But no.
They will not write about the terrible minuses of centralization in the presentations of large Western companies, who really want to sell you multi-million dollar software, and will not say in press releases. But starting such a project, you will get stuck in a long tedious implementation that will last for years. Because the entities that you will bring to one standard have a different nature. And this makes it unreasonable to combine them within a single system.
Do not believe it - look at the table with data types that are usually found in the company:
| What do they describe | Table, machine, etc. | Individuals, legal entities, state of emergency / SP. | Entity attributes: education, profession, etc. |
| Attribute truth criterion | Objective (there is a real machine that you can look at) | Objective (there is a real person or company from whom you can ask everything) | Subjective (expert opinion, which may be kagbe not quite right, or the system) |
| The size | Medium (tens of thousands of units) | Medium or large (hundreds of thousands and millions of records) | Small (thousands of units) |
| Number of Attributes | Secondary (attributes of goods and materials) | Average (customer attributes) | Small (in most cases, a structure of type id is a name) |
| How do existing objects change over time? | Seldom | Constantly changing (change of address, marital status, gender ...) | Unpredictably change depending on the needs of an information system or business process |
In practice, the fastest and most successful implementations take place in those companies that separately solve the goods and materials problem, separately - the task of constructing a directory of counterparties, and somehow deal with classifiers along the way.
Product Information Management (PIM) is usually implemented for goods and materials, Customer Data Integration (CDI) for counterparts, but the interesting thing happens with classifiers: for them, NMI (MDM) is often implemented separately and a single version of the truth is formed.
Is it right to implement NSI and create a single version of classifiers? Or can you solve this problem differently?
Is it good to have a single version of the truth for classifiers?
When classifiers are brought to a single form, one of the scenarios usually occurs:
- A monster reference is created that takes into account all combinations in the source systems. This complex reference book is difficult to modify, requires lengthy coordination, and over time, everything is slowly clogged into it.
- A single directory is created that contains a truncated version of the truth that suits everyone. Those who worked in a large company know that the search for "the truth that suits the swan, cancer and pike" rarely ends successfully, and if it ends, it requires many, many hours of fierce meetings.
In general, a single version of the truth is usually expensive.
In addition, with a single standard, the problem of a “gray zone” always arises. This is the difference between the values in the source systems and the values of the coveted unified version of the truth, which were personally confirmed by a specially appointed expert. The presence of a “gray zone” and the need to work with it leads to the appearance of numerous workflows and alternative flows, which is a significant part of the functionality (read the cost) of the NSI.
And also the business is not always ready to wait for numerous approvals for years. Therefore, in the end, the data begins to be forwarded "under the table", using "temporary" transcodes (what it is, we will describe below).
Is it possible not to create a single version of the truth?
A logical question arises: if a single version of the truth is so expensive, can you do without it?
You can - there are organizations in which NSIs are not implemented. Having carefully looked, you will find in these organizations many small correspondence tables between the reference books “Sex”, “Education”, etc. in different systems. They will be stored either in Oracle, then on the bus, or even directly in the code for download-upload and export-import. Because correspondence tables are the most natural way to solve a problem when the same domain is designated differently for different people or systems.
In other words, you can force all people on earth to speak Esperanto (create a single version of the truth), or you can invite a Chinese-German medical translator to negotiate with Chinese and German doctors (make a correspondence table).
Therefore, we decided to make a simultaneous translator that understands the language of any system and allows them to agree on the same concepts within the framework of the business process. We called the correspondence tables the transcodes, and the system itself - the Transcoder.
The main purpose of the Transcoder is to store copies of directories of the source systems and transcoding between them.
How does it all work
- At the beginning of the work, the Transcoder reads the directories from the databases of the source systems and saves their copies to itself in the database for subsequent automatic synchronization. We work with Sybase, MariaDB, Vertica, MS SQL Server, MySQL, Oracle, PostgreSQL.
Proofpeak - In the Transcoder interface, the operator sets up the correspondence tables of the directory entries. For example, if the male gender in one system is entered as “M” and in the other as “1”, then the operator associates these two values, creating a transcoding.
- Now, any system in the process of exchanging information with another system can request from the Transcoder through the SOAP or REST interface a translation of the value from its classifier into the language of the target system or to translate the received data into its own dialect.
Transcoding from the “Document Type” directory of a certain banking system (ABS) to the “DUL Type” directory for CRM. The passport corresponds to the general foreign passport, the driver's license corresponds to other documents. ABS and CRM do not need to integrate with the master directory, everyone is happy. - The transcoder was originally designed for a load of up to a thousand transcodes per second to satisfy all participants in the information exchange: bus, ETL scripts serving real-time procedures, macros in Excel, space station, etc.
What can I do with the Transcoder
It is possible and necessary:
- Manage directories, reference groups and transcodes (create, edit and delete).
- Set up periodic synchronization of Transcoder directories with source systems.
- Perform online transcoding and get the values of the source directories via SOAP and REST interfaces.
- Receive automatic notifications of changes in source directories. If a value suddenly appears for which there is no recoding, SOAP / REST will return an error and notify all interested parties.
Classifier "Customer Service Program", in which the attributes are for some reason called non-Russian words Description and Full Description
What's under the hood?
The system is divided into modules to isolate the API, business logic and storage from each other. The Spring Framework acts as the connecting IoC container between the components.
The technological scheme of data flows can be represented as follows:
As you can see, services do not interact directly with the database at all. Instead, all the work (read / write) goes with the Lucene index, and the results of the write operations are additionally written to the database. Why is this done? Everything is simple: the main task of the system is to quickly find the transcoding (correspondence between entries in a pair of directories), and such an architecture most effectively copes with this on large amounts of data. Plus, we get a full-text search for all the data “out of the box”. Of course, one could use higher-level solutions, such as Elasticsearch or Solr, but they have their own drawbacks, which are more expensive to fight than writing your own code.
In order to ensure the integrity of information, data modification is performed in one transaction in the search index and in the database. Moreover, the interfaces are designed so that the business logic of the application knows only about the index, and the DBMS can be replaced with another solution with minimal modifications. In fact, you need to implement a couple of classes and add them to the classpath - Spring and the class compilation factories on the fly (runtime code compiling) will do the rest.
The framework for SOAP and RESTfull interfaces is Apache CXF, which makes it easy to configure the version of the API (if you suddenly need to change it) and wedge into any phase of the processing of incoming / outgoing messages.
Source
github.com/hflabs/perecoder
Test stand
rcd.hflabs.ru/rcd/admin/login
Login: admin
Password: demo
Documentation
confluence.hflabs.ru/x/RICSCg
What's next
We started the implementation of the transcoder with several customers.
We welcome your comments on functionality: what you like, what not, what is incomprehensible.
Well, if this product is useful to you, it means that we did not waste our time writing this article;)
