December 1, 2014 at 03:16Software, format, standard and some tediousness - experience in developing a program for processing geobotanical data
Greetings to the venerable Geeks! As a student, I went downhill and became a nerd. Yes, it’s the real one that vomits “grass and lichens,” and then sits with heavy qualifiers. And although this did not become the main type of my activity, I nevertheless got a part of the botanical (or rather geobotanical) work. Under the cut - a look at the work of geobotany from the perspective of an IT specialist, expressed through the experience of developing crutch software. I draw your attention to the fact that the words “geobotany work” and “IT specialist look” located in one sentence mean that the further text is contraindicated for pregnant women, minors, as well as people with unstable psyche and cardiovascular diseases.
People often ask meif I know Tyler Durden who are geobotanists? I answer: geobotanists are specialists in plant communities. Unlike, for example, taxonomists who travel from one dusty herbarium to another, geobotanists collect their material exclusively in the field, compiling geobotanical descriptions - special tables that indicate which plants and in what volumes grow on the area. From these tables in the future, you can determine the indicators of soil fertility and moisture, anthropogenic disturbance, feed value of the site and many other interesting things. And, as you understand, this definition is carried out in the same way as in the era, "when the dinosaurs were small" - with paper form and graph paper. The most advanced use Excel.
To paraphrase Zheglov: “A geobotanic who enters data from a paper form into Excel receives a working card in vain.” And the point here is not at all excel. There are no complaints about this program - it’s a wonderful thing, besides it’s totally stolen, with an available portable version. And even the fact that the xls format is not GOST standard, not only does not bother, but it is not even known to the bulk of specialists. The main problem with translating from paper forms to spreadsheets is the futility of this work.
Why are Excel tables bad? Here's the analogy for books: about five years ago, I began to actively collect a collection of scanned pdf and djvu versions of books. Over the years, the collection has grown to hundreds of gigabytes, and perhaps the only thing in it is not good. After a certain moment, I completely stopped using these books, since it took more time to search for information in my library than to search for such information on the Internet. E-book formats are good for fiction novels on readers, but only the network is suitable for storing technical literature and nothing else. Due to the fact that the "Flora of the USSR" is scanned, it will not become more popular than the botanical sections of Wikipedia.
Same thing with geobotanical descriptions. Working on the project of restored circumboreal vegetation, I collected a small electronic phytocenary (a collection of geobotanical descriptions) - in several hundred descriptions, in a naive attempt to process it. It is required to note here that descriptions in the phytocenary were made in different natural zones, by different authors, at different times, by different methods. Such descriptions are fundamentally impossible to standardize and compare, unless of course we are talking about rough qualitative comparisons (“there were lichens, and there were mosses and ewon berries”). Even the work of creating a single database on the basis of these descriptions is painful and therefore wrong.
Yes, I again need (or do I need?) The lack of standards and the absurdity of modern geobotany. This Leonty Ramensky could afford to collect tens of thousands of descriptions and manually process them. Today it is impossible - nobody needs such work, even though it is really possible to increase productivity due to technical means. Therefore, if we want to work with large phytocenarios, it is necessary to combine the achievements of each in a single base. But for this, one should at least draw up descriptions according to uniform standards, and not as necessary.
Yes, I certainly understand that “scientific” field work is today very often nothing more than paid tourist trips. Therefore, the question of uniform standards and methods is not raised. Therefore, the question of the advisability of publishing descriptions in the journal “Vegetation of Russia” does not arise (for the publication of tables of descriptions in the PAPER magazine it is high time to give the order of “honorary starter”). However, in the courtyard of the 21st century, geobotanists continue to fill out paper forms of their own invention.
The first attempt to optimize the work with raw descriptions I made three years ago as part of the PhytoSoft program (development was carried out in Borland C ++ Builder 6). At that time, the task was to facilitate and speed up data entry from the field form, for subsequent analysis on environmental scales of L.G. Ramensky (remember, I said earlier that with the help of geobotanical descriptions it is possible to determine the fertility and moisture of the soil? This is precisely the "method of ecological scales"). The program was able to bring to a healthy state, but with an extremely low budget, it remained at the alpha testing stage and was later laid out in the public domain with all its cockroaches.
Now I understand that the concept of "Fitosoft" contained a few terrible strategic mistakes. And it's not even that the code is (cut out by censorship) and the hands are growing from the same place as the legs. The very idea of simplifying the entry of descriptions from the form is fundamentally wrong.
While showing the first successfully compiled version, I regularly heard a question about the possibility of importing descriptions from Excel into Fitosoft. Undoubtedly, during the development I foresaw such an opportunity, but the import technology was ugly and I always tried to silence this issue, although it was and remains one of the main ones. Even if now, a miracle program for geobotanists will appear, what to do with those descriptions that are already entered into table editors? As I said above, it’s practically impossible to bring descriptions of different authors to a single template, respectively, each description will either have its own structure, or the general structure will be super-complex and there will always be a description that does not fit into this structure. A fundamentally different approach was needed to organize the data in the descriptions than those to which we are accustomed (line-columns).
The * .gbo format that was used in Phytosoft did not fit this task in any way. I did not properly design the specification for it, but most importantly, it also represented the very same “line-column”. Simply put, "* .gbo" is a huge table, one thousand rows high, several hundred columns wide. Each description in the table occupies one line. The description is divided into logical elements that are placed in different cells. For example, the author of the first description is indicated in the fifth cell of the first line, the date of the first description is indicated in the sixth cell of the first line, etc., the author of the second description is indicated in the fifth cell of the second line, the date of the second description is indicated in the sixth cell of the second line, etc. ... The logic of the format is very simple, but to import external files, the latter had to be painfully redone (imagine: your summary description table needs to be rebuilt in such a way that the names of the species go from column 50 to 100, and projective coverings from 101 to 151). This problem arose from the fact that instead of developing a program for a format, there was a development of a format for a program.
Maybe I invented a bicycle, but on my own skin I realized that software and file format are not related (in terms of development) things. Initially, you should develop a file format, and do it regardless of when and by whom the software will be written for this format. After that, it already makes sense to write a program. At the same time, it will be necessary to solve many problems that would not have arisen when creating the format “for yourself,” but on the other hand, the risks that the format will have critical drawbacks are reduced.
Here is a concrete example. If you are initially developing the format as a separate project, you will probably consider that it should be adapted for import into GIS. After that, starting development, you will be forced to solve the problem with GIS compatibility, even if the geographic information systems in your program do not smell. But they will not refuse this format, which would inevitably happen when it turned out that the format is useless for ArcGIS, QGIS or another program.
The file format should be as convenient as possible for converting to and from other formats.
When developing Fitosoft as part of the idea of simplifying the digitization of paper forms, I was mistaken. There should not be paper forms at all; field data should be immediately ready for processing. But, since any dead end path sooner or later ends, the applied concept quickly outlived itself, giving rise to a new problem. On the one hand, it became clear that the geobotanist should enter data into the program immediately after receiving it in the field. On the other hand, this means that he will not use either a computer or a laptop. Tablets? But their price is as immoral as the battery life. And most importantly, I remembered in what condition the field forms come back - wet, with blood stains from crushed mosquitoes. And on the part of the user, I would not want that my (and not only mine) habit, without hesitation, throwing a tablet (which fastens paper forms) next to itself, once ruined the electronics. The tablet did not fit. There was a smartphone. From the point of view of use, this is ideal - they cost much less, take them with you in any weather and the likelihood of damage to them is much less. But at the same time, this means that all code associated with the Fitosoft interface can be deleted. For field conditions, something fundamentally different than a lot of selection windows is required.
The organization of a new interface has long seemed a dead end and at present there is still no complete clarity in some details. As the format is developed, it becomes obvious that it should be as simple and compact as possible. As far as I imagine, this should be something similar to the command line. As regards the format, I consider it most reasonable to abandon the tabular organization of data in favor of an HTML-like metalanguage. On the one hand, this will make it possible to easily restore tabular descriptive structures, on the other hand, descriptions already entered into tabular editors are much easier to bring to the standard by adding appropriate tags.
For sim, I have exhausted my story. And with gratitude I will accept any criticism and advice - for, as one cannibal liked to repeat: "One head is good, and two is better."
“Who is Tyler Durden?”
People often ask me
“And now, Brokeback!”
To paraphrase Zheglov: “A geobotanic who enters data from a paper form into Excel receives a working card in vain.” And the point here is not at all excel. There are no complaints about this program - it’s a wonderful thing, besides it’s totally stolen, with an available portable version. And even the fact that the xls format is not GOST standard, not only does not bother, but it is not even known to the bulk of specialists. The main problem with translating from paper forms to spreadsheets is the futility of this work.
Why are Excel tables bad? Here's the analogy for books: about five years ago, I began to actively collect a collection of scanned pdf and djvu versions of books. Over the years, the collection has grown to hundreds of gigabytes, and perhaps the only thing in it is not good. After a certain moment, I completely stopped using these books, since it took more time to search for information in my library than to search for such information on the Internet. E-book formats are good for fiction novels on readers, but only the network is suitable for storing technical literature and nothing else. Due to the fact that the "Flora of the USSR" is scanned, it will not become more popular than the botanical sections of Wikipedia.
Same thing with geobotanical descriptions. Working on the project of restored circumboreal vegetation, I collected a small electronic phytocenary (a collection of geobotanical descriptions) - in several hundred descriptions, in a naive attempt to process it. It is required to note here that descriptions in the phytocenary were made in different natural zones, by different authors, at different times, by different methods. Such descriptions are fundamentally impossible to standardize and compare, unless of course we are talking about rough qualitative comparisons (“there were lichens, and there were mosses and ewon berries”). Even the work of creating a single database on the basis of these descriptions is painful and therefore wrong.
“We must understand the whole depth of our depths!”
Yes, I again need (or do I need?) The lack of standards and the absurdity of modern geobotany. This Leonty Ramensky could afford to collect tens of thousands of descriptions and manually process them. Today it is impossible - nobody needs such work, even though it is really possible to increase productivity due to technical means. Therefore, if we want to work with large phytocenarios, it is necessary to combine the achievements of each in a single base. But for this, one should at least draw up descriptions according to uniform standards, and not as necessary.
Yes, I certainly understand that “scientific” field work is today very often nothing more than paid tourist trips. Therefore, the question of uniform standards and methods is not raised. Therefore, the question of the advisability of publishing descriptions in the journal “Vegetation of Russia” does not arise (for the publication of tables of descriptions in the PAPER magazine it is high time to give the order of “honorary starter”). However, in the courtyard of the 21st century, geobotanists continue to fill out paper forms of their own invention.
"This is not for you!"
The first attempt to optimize the work with raw descriptions I made three years ago as part of the PhytoSoft program (development was carried out in Borland C ++ Builder 6). At that time, the task was to facilitate and speed up data entry from the field form, for subsequent analysis on environmental scales of L.G. Ramensky (remember, I said earlier that with the help of geobotanical descriptions it is possible to determine the fertility and moisture of the soil? This is precisely the "method of ecological scales"). The program was able to bring to a healthy state, but with an extremely low budget, it remained at the alpha testing stage and was later laid out in the public domain with all its cockroaches.
Now I understand that the concept of "Fitosoft" contained a few terrible strategic mistakes. And it's not even that the code is (cut out by censorship) and the hands are growing from the same place as the legs. The very idea of simplifying the entry of descriptions from the form is fundamentally wrong.
While showing the first successfully compiled version, I regularly heard a question about the possibility of importing descriptions from Excel into Fitosoft. Undoubtedly, during the development I foresaw such an opportunity, but the import technology was ugly and I always tried to silence this issue, although it was and remains one of the main ones. Even if now, a miracle program for geobotanists will appear, what to do with those descriptions that are already entered into table editors? As I said above, it’s practically impossible to bring descriptions of different authors to a single template, respectively, each description will either have its own structure, or the general structure will be super-complex and there will always be a description that does not fit into this structure. A fundamentally different approach was needed to organize the data in the descriptions than those to which we are accustomed (line-columns).
The * .gbo format that was used in Phytosoft did not fit this task in any way. I did not properly design the specification for it, but most importantly, it also represented the very same “line-column”. Simply put, "* .gbo" is a huge table, one thousand rows high, several hundred columns wide. Each description in the table occupies one line. The description is divided into logical elements that are placed in different cells. For example, the author of the first description is indicated in the fifth cell of the first line, the date of the first description is indicated in the sixth cell of the first line, etc., the author of the second description is indicated in the fifth cell of the second line, the date of the second description is indicated in the sixth cell of the second line, etc. ... The logic of the format is very simple, but to import external files, the latter had to be painfully redone (imagine: your summary description table needs to be rebuilt in such a way that the names of the species go from column 50 to 100, and projective coverings from 101 to 151). This problem arose from the fact that instead of developing a program for a format, there was a development of a format for a program.
“This is not a tub, but a real Japanese furake!”
Maybe I invented a bicycle, but on my own skin I realized that software and file format are not related (in terms of development) things. Initially, you should develop a file format, and do it regardless of when and by whom the software will be written for this format. After that, it already makes sense to write a program. At the same time, it will be necessary to solve many problems that would not have arisen when creating the format “for yourself,” but on the other hand, the risks that the format will have critical drawbacks are reduced.
Here is a concrete example. If you are initially developing the format as a separate project, you will probably consider that it should be adapted for import into GIS. After that, starting development, you will be forced to solve the problem with GIS compatibility, even if the geographic information systems in your program do not smell. But they will not refuse this format, which would inevitably happen when it turned out that the format is useless for ArcGIS, QGIS or another program.
The file format should be as convenient as possible for converting to and from other formats.
When developing Fitosoft as part of the idea of simplifying the digitization of paper forms, I was mistaken. There should not be paper forms at all; field data should be immediately ready for processing. But, since any dead end path sooner or later ends, the applied concept quickly outlived itself, giving rise to a new problem. On the one hand, it became clear that the geobotanist should enter data into the program immediately after receiving it in the field. On the other hand, this means that he will not use either a computer or a laptop. Tablets? But their price is as immoral as the battery life. And most importantly, I remembered in what condition the field forms come back - wet, with blood stains from crushed mosquitoes. And on the part of the user, I would not want that my (and not only mine) habit, without hesitation, throwing a tablet (which fastens paper forms) next to itself, once ruined the electronics. The tablet did not fit. There was a smartphone. From the point of view of use, this is ideal - they cost much less, take them with you in any weather and the likelihood of damage to them is much less. But at the same time, this means that all code associated with the Fitosoft interface can be deleted. For field conditions, something fundamentally different than a lot of selection windows is required.
"I'll be back"
The organization of a new interface has long seemed a dead end and at present there is still no complete clarity in some details. As the format is developed, it becomes obvious that it should be as simple and compact as possible. As far as I imagine, this should be something similar to the command line. As regards the format, I consider it most reasonable to abandon the tabular organization of data in favor of an HTML-like metalanguage. On the one hand, this will make it possible to easily restore tabular descriptive structures, on the other hand, descriptions already entered into tabular editors are much easier to bring to the standard by adding appropriate tags.
For sim, I have exhausted my story. And with gratitude I will accept any criticism and advice - for, as one cannibal liked to repeat: "One head is good, and two is better."
