April 28, 2012 at 04:07Data collection from recording devices
Greetings to all the inhabitants of the Habr! I would like to share my idea, which I partially managed to realize. But I can’t appreciate the significance of this kind of project, and I would like to hear your opinion and your constructive criticism about this.
Having arrived at the native plant as an engineer in the ASODU group, it was one of my tasks to accompany the mechanism for collecting data from various types of recording devices. I note that the plant has a good "zoo" of this kind of equipment. As you know, for recording devices, there is always specialized software that allows you to configure and poll it. But far from all types of devices, there is software that can be used to obtain data from the device and place it in a common environment for further processing and archiving. So this problem was solved before me, by writing one program that would query all the devices available, and uploading the collected data to a single database. But the problem was that with the advent of a new type of device, it was constantly necessary to recompile this program, in addition, it is tightly attached to a specific DBMS. Not having at hand either a configurator or any tester, all this led the tracking process to continuous torment. And then the idea came up to implement a certain system that would simplify the work of maintaining the data collection mechanism as much as possible. For such a system, I put forward the following requirements:
To implement my idea, I used the following tools:
Compilers: gcc-3.4.2, gcc-4.6.1 and tinyc-0.9.25
Graphic library: wxWidgets-1.8.10 + wxFormBuilder-3.2
Database: SQLite-3.7.6.2
In the implementation, I will briefly talk only about the main part of the program - data collection. The remaining parts of the software package, in my opinion, are not as interesting as this one in their implementation.
The most important task for me was to store data in the system. The buffer implemented by me took the following form:
Each poll stream has access to its own buffer branch, with which it takes all the start (initializing) settings, writes its status and enters the received data. The data upload stream has access to both the poll branch (with read-only access) and its own export branch. Access to objects marked in red in the picture is synchronized using critical sections.
The core of the program is shown in the figure below. It works as follows. When reading the configuration file, the loader forms the final buffer (see the figure above), the list of downloaded polling plugins, and the list of data upload plugins. The kernel receives both the generated buffer and the lists of plugins. The kernel, in turn, initializes the stream to each com-port, while passing it a buffer branch and a list of device plug-ins. Once the polling streams are all created, the kernel starts to initialize the data export streams in the same way. But, besides the ones listed above, it passes to everyone a constant link to the poll branch. Thus, export flows can at any time receive all the information, both on the progress of the survey process, and obtain the final results of the survey.
Since there are two types of polling programs (as a service and as a custom graphical application), the kernel is placed in a dynamic library that both applications use.
All threads generated by the kernel, among other things, get a constant reference to the kernel itself. Thus, each of the threads can monitor the state of the kernel (RUN, STOP). In the event that the kernel enters STOP mode, then all threads begin to automatically shut down. When switching to RUN mode, the kernel re-creates the threads described above.
In order to interrogate a device, it is necessary to have two functions: a function that forms the final packet sent to the device, and a function that processes the received response from the device. Thus, the plugin interface consists of two functions of forming and processing the package and another function that returns information about it. This information is needed both by the user and the program, due to the fact that its contents contain information about the length of the generated and sent packets. As a result, we have the following functions:
The structures that operate these functions, I think it is unnecessary to describe, because my goal is to describe the general principle of the system. But to whom it became interesting, can look into the documentation .
The export plugin interface consists of four functions:
The structures that operate these functions, I also will not describe because of the above.
At the moment, the complex includes the following software:
The complex is focused on Windows, although a tight binding to WinAPI is available only in a class that works with a COM port. Well, of course, the device polling service. Everything else is based on the classes and functions of the wxWidgets library.
Suppose there are two devices of the rmt-59 and ecograph-t type . Each of them is connected to a separate port to the interface converter “RS-485 - Ethernet”. On the computer that performs the survey, there is a driver that converts “Ethernet - RS232”. Thus, we have two com-ports (for example com-10 and com-11), on which one device is located. For both devices, suppose address 1. Both devices are configured for a data rate of 19,200 bps.
First you need to make sure that the available plug-ins are suitable for working with these devices. To do this, run the TestRequest program and try to interrogate these devices
After that, you should create a survey configuration. Run the Editor program and set up the survey.
If you created a new database, then you need to register the path to it in the Reader.ini file. To test the operation of the survey run the program ReaderGUI
Now, you need to take care of exporting the data. I did not create specialized plugins. For the test of export work, the kit includes one test plug-in that exports data to a text file. First, you should check its operation using the TestExport program.
Now that we have verified that the plugin is working correctly, we can add it to the survey configuration.
Everything, configuration and testing is completed. Now you can install and start the service. We manage the service using the ReaderSvcCtrl program.
Of course, I could write much more, but in order not to bore the reader, I will not do this. Here is a link to my project. I will be happy to answer all your questions in the comments.
Many disadvantages of the system were found out during operation. These disadvantages include the following:
Brief Background
Having arrived at the native plant as an engineer in the ASODU group, it was one of my tasks to accompany the mechanism for collecting data from various types of recording devices. I note that the plant has a good "zoo" of this kind of equipment. As you know, for recording devices, there is always specialized software that allows you to configure and poll it. But far from all types of devices, there is software that can be used to obtain data from the device and place it in a common environment for further processing and archiving. So this problem was solved before me, by writing one program that would query all the devices available, and uploading the collected data to a single database. But the problem was that with the advent of a new type of device, it was constantly necessary to recompile this program, in addition, it is tightly attached to a specific DBMS. Not having at hand either a configurator or any tester, all this led the tracking process to continuous torment. And then the idea came up to implement a certain system that would simplify the work of maintaining the data collection mechanism as much as possible. For such a system, I put forward the following requirements:
- Interview any type / type of instrument. This should be achieved by expanding the program through modules.
- Unload data as you like, as you like and anytime. This approach should also be achieved through modules.
- The presence of a tool that makes it possible to easily configure the entire survey process.
- The presence of a tool that allows you to test and debug both polling modules and data upload modules.
- The task of the survey should work as a service, but at the same time, the ability to visually monitor the progress of the survey and upload the data should also be present.
Tools
To implement my idea, I used the following tools:
Compilers: gcc-3.4.2, gcc-4.6.1 and tinyc-0.9.25
Graphic library: wxWidgets-1.8.10 + wxFormBuilder-3.2
Database: SQLite-3.7.6.2
Implementation
In the implementation, I will briefly talk only about the main part of the program - data collection. The remaining parts of the software package, in my opinion, are not as interesting as this one in their implementation.
Clipboard
The most important task for me was to store data in the system. The buffer implemented by me took the following form:
Each poll stream has access to its own buffer branch, with which it takes all the start (initializing) settings, writes its status and enters the received data. The data upload stream has access to both the poll branch (with read-only access) and its own export branch. Access to objects marked in red in the picture is synchronized using critical sections.
Core
The core of the program is shown in the figure below. It works as follows. When reading the configuration file, the loader forms the final buffer (see the figure above), the list of downloaded polling plugins, and the list of data upload plugins. The kernel receives both the generated buffer and the lists of plugins. The kernel, in turn, initializes the stream to each com-port, while passing it a buffer branch and a list of device plug-ins. Once the polling streams are all created, the kernel starts to initialize the data export streams in the same way. But, besides the ones listed above, it passes to everyone a constant link to the poll branch. Thus, export flows can at any time receive all the information, both on the progress of the survey process, and obtain the final results of the survey.
Since there are two types of polling programs (as a service and as a custom graphical application), the kernel is placed in a dynamic library that both applications use.
All threads generated by the kernel, among other things, get a constant reference to the kernel itself. Thus, each of the threads can monitor the state of the kernel (RUN, STOP). In the event that the kernel enters STOP mode, then all threads begin to automatically shut down. When switching to RUN mode, the kernel re-creates the threads described above.
Device Polling Plugin Interface
In order to interrogate a device, it is necessary to have two functions: a function that forms the final packet sent to the device, and a function that processes the received response from the device. Thus, the plugin interface consists of two functions of forming and processing the package and another function that returns information about it. This information is needed both by the user and the program, due to the fact that its contents contain information about the length of the generated and sent packets. As a result, we have the following functions:
- GetInfo - plugin information;
- GetPackage - package formation;
- GetData - processing the received packet.
The structures that operate these functions, I think it is unnecessary to describe, because my goal is to describe the general principle of the system. But to whom it became interesting, can look into the documentation .
Export Plugin Interface
The export plugin interface consists of four functions:
- About - information about the plugin;
- Begin - the function is executed once, after the kernel enters the RUN mode. It is focused on making any connection to the data warehouse. If the function returns an error, the thread writes the error to the buffer and exits.
- Export - directly export data;
- End - the function is executed once, after the kernel enters the STOP mode. It is executed only if the Begin function did not return any error. The function is designed to disconnect from the data warehouse.
The structures that operate these functions, I also will not describe because of the above.
Result
At the moment, the complex includes the following software:
- Editor - poll configuration editor;
- ReaderGUI - a polling program in the form of a user application;
- ReaderSvc - a polling program in the form of a Windows service;
- ReaderSvcCtrl - polling service management;
- TestExport - testing export plugins;
- TestRequest - Testing survey plugins.
The complex is focused on Windows, although a tight binding to WinAPI is available only in a class that works with a COM port. Well, of course, the device polling service. Everything else is based on the classes and functions of the wxWidgets library.
Work example
Suppose there are two devices of the rmt-59 and ecograph-t type . Each of them is connected to a separate port to the interface converter “RS-485 - Ethernet”. On the computer that performs the survey, there is a driver that converts “Ethernet - RS232”. Thus, we have two com-ports (for example com-10 and com-11), on which one device is located. For both devices, suppose address 1. Both devices are configured for a data rate of 19,200 bps.
First you need to make sure that the available plug-ins are suitable for working with these devices. To do this, run the TestRequest program and try to interrogate these devices
After that, you should create a survey configuration. Run the Editor program and set up the survey.
If you created a new database, then you need to register the path to it in the Reader.ini file. To test the operation of the survey run the program ReaderGUI
Now, you need to take care of exporting the data. I did not create specialized plugins. For the test of export work, the kit includes one test plug-in that exports data to a text file. First, you should check its operation using the TestExport program.
Now that we have verified that the plugin is working correctly, we can add it to the survey configuration.
Everything, configuration and testing is completed. Now you can install and start the service. We manage the service using the ReaderSvcCtrl program.
Afterword
Of course, I could write much more, but in order not to bore the reader, I will not do this. Here is a link to my project. I will be happy to answer all your questions in the comments.
Many disadvantages of the system were found out during operation. These disadvantages include the following:
- Lack of type of the polled channel. The necessity of the type is due to the fact that many devices, for different types of values (analog channel, mathematical channel, integral value), require a specially formed request.
- Lack of a multiplier concept. Those. some devices, when polled, transmit the value as an integer. And the position of the comma in the number, for such devices, is interrogated separately. And having such a multiplier, the user could change the position of the comma. For example, if on a given channel the separator between an integer and a fractional value is placed after the first number, then a factor of 0.1 would allow you to bring the number in the proper form. And having received the value 123, the system, multiplying this number by the corresponding factor, would give the result 12.3.
