May 9, 2019 at 12:14Experience in bringing the software implementation of the social project “Remember Everyone” to open source
Last year, our team was lucky to take part in the development of software for a socially significant project - a system for entering archive data from the military commissariats of the Khabarovsk Territory during the Great Patriotic War. In short, when he was called up, where did he go and other related information - to digitize and give anyone the opportunity to search for this data. In 2018, the Far Eastern Center for Social Technologies, with the support of the Presidential Grants Fund, implemented the project “Remember Everyone”. And we have developed an application on our open source product IONDV. Framework The final application is now available under the GPLv3 license.
We share our decision and experience on the project.
Spoiler, some technical details.
During development, we created a web application, with the corporate name War archive, for storing, grouping and demonstrating archive documents about the Great Patriotic War. We worked actively from January to March 2018 and in the process we fixed the framework bugs - as we had the first experience with so many scans.
The result of the project “Remember Everyone” was a data register with 25k records (now already 35k). In April this year, the Far Eastern Center for Social Technologies published the source code for the application under the GPLv3 license on GitHub . In the repository you will find metadata, design templates and special utilities - everything that forms the basis of the application for IONDV. Framework The framework itself is licensed under Apache 2.0 and is also available on GitHub., together with ready-made modules.
Also deployed a demo application . Test query to search for "Ivanov Ivan." You can see the backoffice here . Login - demo, password - ion-demo. By the way, you can get a ready-made docker image .
The goal of the “Remember Everyone” project is to preserve the memory of people from the time of the Second World War by providing free access to documents of those years. Maria Stepko, director of the Far Eastern Center for Social Technologies once said: “The preservation and analysis of the data on the past is a task necessary to model the future.”
Volunteers of the project digitized 10 thousand pages of documents and entered data into the register about the Far East, participants in the Great Patriotic War. In the database you can find information about the rank of the called, about the part, date, cause of death and the place of burial. In addition, a digitized source of records is available indicating the number of inventory, case, sheet in accordance with the numbering of the state archive. An example of a document scan.
We are very grateful for this opportunity and for feedback on our work:
Yesterday we finished transferring the project to Yandex. The cloud was made available at the new address dvarchive.ru .
At the same time, we conducted stress testing, which we would also like to share.
Server resources are minimal, as it is paid for by the fundraising company: 2 CPUs with 20% priority and 2Gb of memory.
A typical reaction to a light load looks like this - 2% of the CPU load and 36% of the memory load.
We tested user requests without static files, only data requests and we liked the result.
It turned out that the application in the docker container processes 400 requests per second (without statics). And at the same time it loads only one processor, which is typical when starting only one instance of node.js. The second handles the rest of the tasks (DBMS, nginx, monitoring).
Do not even need clustering. There is a large margin for the usual workload of a social project, despite the minimum resources.
For us, this was the first experience of participating in the work on a project related to the memory of the Great Patriotic War and the Far East.
Moreover, we are proud that this result, developed in the Far East, a region that is not very active in the field of IT, can be used by anyone involved in such projects, especially according to the open source model.
We share our decision and experience on the project.
Spoiler, some technical details.
IONDV. Framework
IONDV. Framework - open source framework on node.js for creating high-level metadata-based web applications, which does not require serious programming skills.
The basis of the application functionality is the data registry - the Register module. This is a key module designed specifically for working with data based on metadata structures - including those for managing projects, programs, events, etc. The project also uses a portal module for displaying arbitrary data templates - it implements the front of the registry of archive files.
MongoDb is used for the DBMS - it also stores application settings, metadata and the data itself.
The basis of the application functionality is the data registry - the Register module. This is a key module designed specifically for working with data based on metadata structures - including those for managing projects, programs, events, etc. The project also uses a portal module for displaying arbitrary data templates - it implements the front of the registry of archive files.
MongoDb is used for the DBMS - it also stores application settings, metadata and the data itself.
During development, we created a web application, with the corporate name War archive, for storing, grouping and demonstrating archive documents about the Great Patriotic War. We worked actively from January to March 2018 and in the process we fixed the framework bugs - as we had the first experience with so many scans.
The result of the project “Remember Everyone” was a data register with 25k records (now already 35k). In April this year, the Far Eastern Center for Social Technologies published the source code for the application under the GPLv3 license on GitHub . In the repository you will find metadata, design templates and special utilities - everything that forms the basis of the application for IONDV. Framework The framework itself is licensed under Apache 2.0 and is also available on GitHub., together with ready-made modules.
Also deployed a demo application . Test query to search for "Ivanov Ivan." You can see the backoffice here . Login - demo, password - ion-demo. By the way, you can get a ready-made docker image .
Project Details
The goal of the “Remember Everyone” project is to preserve the memory of people from the time of the Second World War by providing free access to documents of those years. Maria Stepko, director of the Far Eastern Center for Social Technologies once said: “The preservation and analysis of the data on the past is a task necessary to model the future.”
Volunteers of the project digitized 10 thousand pages of documents and entered data into the register about the Far East, participants in the Great Patriotic War. In the database you can find information about the rank of the called, about the part, date, cause of death and the place of burial. In addition, a digitized source of records is available indicating the number of inventory, case, sheet in accordance with the numbering of the state archive. An example of a document scan.
We are very grateful for this opportunity and for feedback on our work:
“The technical solution created by the Far Eastern development company IONDV goes into free distribution. The system is suitable for NGOs, initiative citizens, museums, archives. It will help to save resources and do a good deed for people. Programmers in our region know what “free software” is. Together, we will make this software more functional.
Deployment and Test Result
Yesterday we finished transferring the project to Yandex. The cloud was made available at the new address dvarchive.ru .
At the same time, we conducted stress testing, which we would also like to share.
Server resources are minimal, as it is paid for by the fundraising company: 2 CPUs with 20% priority and 2Gb of memory.
A typical reaction to a light load looks like this - 2% of the CPU load and 36% of the memory load.
We tested user requests without static files, only data requests and we liked the result.
It turned out that the application in the docker container processes 400 requests per second (without statics). And at the same time it loads only one processor, which is typical when starting only one instance of node.js. The second handles the rest of the tasks (DBMS, nginx, monitoring).
Do not even need clustering. There is a large margin for the usual workload of a social project, despite the minimum resources.
Total
For us, this was the first experience of participating in the work on a project related to the memory of the Great Patriotic War and the Far East.
Moreover, we are proud that this result, developed in the Far East, a region that is not very active in the field of IT, can be used by anyone involved in such projects, especially according to the open source model.