January 17, 2017 at 09:20Second Generation Cloud Computing: CLAVIRE System
Under the cut in this article is a story about what AaaS is, why the Application as a Service model is considered more promising than the familiar SaaS, PaaS and IaaS, and how ITMO University developers and scientists implement such a model in the framework of the CLAVIRE project. Torley / the Flickr / CC
Today, cloud computing technologies give the user the opportunity to get network access to an array of information resources (networks, remote servers, applications and services), which can be quickly reserved and provided by the provider with minimal effort.
Developed cloud environments are characterized by the integration of various business models, which is associated with orientation to the specific needs of users in different fields of activity. For their classification and analysis, the Cloud Computing Maturity Model (CCMM) is used, which includes five levels of development presented in the diagram below.
Cloud technologies of I and II generation in CCMM logic. Source : Second Generation Cloud Computing: Composite Applications, Interactive Systems, and Semantic Technologies / A.V. Bukhanovsky, S.V. Kovalchuk
According to CCMM, cloud computing of the first and second generations is distinguished. First-generation technologies correspond to levels I – III and cover three well-known business models - IaaS, PaaS, SaaS. Second-generation technologies include levels IV and V and are focused not only on the provision of individual services to users over the Internet, but also on the management of an entire ecosystem of computing, software and information resources.
This allows you to solve complex problems associated with the joint activities of several users. Second-generation cloud computing technologies implement the promising AaaS (Application as a Service) cloud computing model, the main service of which is the development and use of composite applications (this is a combination of interacting cloud services aimed at solving one problem).
This concept is embodied in the CLAVIRE (CLoud Application VIRtual Environment) multifunctional instrumental technology platform (MITP) developed at the Research Institute for High-Tech Computer Technologies of ITMO University as part of the project for implementing the resolutionGovernment of the Russian Federation, associated with the support of the development of cooperation of higher educational institutions and organizations.
The main tasks of MITP are the effective management of computing, information and software resources of distributed heterogeneous computing infrastructures, as well as the construction of corporate cloud infrastructures.
The CLAVIRE platform provides access to application packages that operate on remote computing resources. Currently, the CLAVIRE package database includes 65 software complexes in such areas as hydroaerodynamics, nanotechnology and quantum chemistry, analysis and modeling of social systems and transport infrastructure, hydrometeorology, shipbuilding and bioinformatics (this means that a number of complex multi-factorial solutions can already be solved on the CLAVIRE platform tasks in these areas - for example, to predict the likelihood of flood surges in St. Petersburg).
In addition, CLAVIRE provides multi-purpose calculations in SciLab, WEKA packages (demo packages available for public use through CLAVIRE interfaces can be found atlink ).
The CLAVIRE platform also allows you to design composite applications - complexes that interact to solve a common problem. A collection of composite applications for various subject areas was provided by the specialists of the Research Institute for High-Tech Computer Technologies of NRU ITMO and today contains several models.
Among them, it should be noted an application for calculating the distribution of temperature and mass velocities in the volume of the convection layer, a composite application for calculating the pitching characteristics of a vessel, and an application for solving the problem of calculating the dynamics of HIV spread.
The solution to the problem of calculating the distribution of temperature and mass velocities in the volume of the convection layer is based on modeling the dynamics of nanoparticles in a medium using the molecular dynamics method and numerically modeling the convection process using the finite element method (packages used: VMD, NAMD, Comsol, MDlogAnalyst, ComsolInitialise, Ruby interpreter)
The calculations themselves are performed on the capacities of Amazon EC2, OpenStack, using grid computing. Grid computing is a type of parallel computing that is performed using conventional computer systems connected to a network and combined into a “virtual supercomputer”.
The CLAVIRE system uses the GridNNS project (a grid system for the National Nanotechnology Network implemented in Russian supercomputer centers). The grid network of supercomputers is controlled by basic grid services located in the primary and backup control centers, the task of which is to coordinate the work of system resources and at the same time be a layer between users and the grid infrastructure.
Computing infrastructure. Source : CLAVIRE Cloud computing platform for high-performance computing / A.V. Bukhanovsky
To control the launch of tasks on multiprocessor installations, solutions such as PBS, Torque, Windows HPC, and Condor are used. Access to resources is via SSH or through web services, and Windows and Unix-like systems are used as operating systems. As for virtualization tools, CLAVIRE uses VirtualBox applications and VMware and KVM solutions.
The organization of the process of creating and executing a composite application under the control of MITP is reduced to the sequential formalization of sets of descriptions in terms of job flows (workflow, WF). At the top level of the application description is meta-WF. In it, individual blocks contain only instructions for performing design tasks.
Thus, MWF is a formal description of the user task in terms of the subject area, without indicating the conditions for its implementation. In addition to describing the actions and data necessary for the calculations, the user has the ability to set criteria and set restrictions by which resources and specific services will be selected.
Composite application in the form of WF. Source : Cloud Platform for High Performance Computing / A.V. Bukhanovsky, S.V. Kovalchuk
Designing a composite application with input data is a process of step-by-step refinement of MWF through the stages of abstract (AWF) and specific WF (CWF), up to the creation of specific scenarios for launching services in the cloud. At the first stage of designing a composite application, MWF is formed. The user is able to choose the classes of services that will be used when selecting services.
Schematic diagram of the creation and execution of a composite application. Source : CLAVIRE: Promising second-generation cloud computing technology / A.V. Bukhanovsky, V.N. Vasiliev, V.N. Vinogradov et al.
Next, WF is formed, in which specific implementations of computing services will be recorded. The next stage of design is scheduling and execution script in terms of CWF. For action blocks, services and nodes for execution are indicated, and for data blocks, the specific location of information.
The figure above shows a schematic diagram of the creation and execution of a composite application managed by MITP.
As an example of CLAVIRE's work, one can cite a system simulating the behavior of crowds in a stadium during the World Cup or other public events. The model takes into account the characteristics of the crowd, such as its social structure, and external factors, such as weather and political conditions. As a result, scientists see a clear picture of the behavior of people in given conditions:
Using the software system, you can recreate any territory and external conditions, and then place a certain group of people in this environment. This opens up tremendous opportunities for research. You can, for example, simulate an earthquake in a particular seismically dangerous region during a holiday, when thousands of people are on the streets of the city.
Sergei Ivanov, head of the international laboratory City Informatics, ITMO University, explained that for the basic operation of the models, ordinary outdoor surveillance cameras at different points in the territory are enough. This will allow you to adjust the model to the real situation, simply by highlighting groups of people from the video sequence and projecting them onto the model.
Field studyusing technology, it was recently conducted by ITMO University staff together with colleagues from other countries. The object of research was the religious festival "Kumbh Mela" in India. Within a month, researchers observed the pilgrimage of about 70 million people to holy places.
Scopes of MITP. Source : Cloud Platform for High Performance Computing / A.V. Bukhanovsky, S.V. Kovalchuk
The complex finds its application in medicine, when its task is to improve the health care system. Decision support systems become software assistants to doctors at all stages of work with patients. They allow you to summarize a huge body of data on the best medical practices and develop more effective methods of treatment and surgery.
PS If you are interested in the system, then a set of technical documentation, methods for using the CLAVIRE complex, program and operational documents, and initial skills for working with the platform can be obtained here . You can get to know CLAVIRE better and work with the system yourself here .
AaaS: second generation cloud technology
Today, cloud computing technologies give the user the opportunity to get network access to an array of information resources (networks, remote servers, applications and services), which can be quickly reserved and provided by the provider with minimal effort.
Developed cloud environments are characterized by the integration of various business models, which is associated with orientation to the specific needs of users in different fields of activity. For their classification and analysis, the Cloud Computing Maturity Model (CCMM) is used, which includes five levels of development presented in the diagram below.
Cloud technologies of I and II generation in CCMM logic. Source : Second Generation Cloud Computing: Composite Applications, Interactive Systems, and Semantic Technologies / A.V. Bukhanovsky, S.V. Kovalchuk
According to CCMM, cloud computing of the first and second generations is distinguished. First-generation technologies correspond to levels I – III and cover three well-known business models - IaaS, PaaS, SaaS. Second-generation technologies include levels IV and V and are focused not only on the provision of individual services to users over the Internet, but also on the management of an entire ecosystem of computing, software and information resources.
This allows you to solve complex problems associated with the joint activities of several users. Second-generation cloud computing technologies implement the promising AaaS (Application as a Service) cloud computing model, the main service of which is the development and use of composite applications (this is a combination of interacting cloud services aimed at solving one problem).
This concept is embodied in the CLAVIRE (CLoud Application VIRtual Environment) multifunctional instrumental technology platform (MITP) developed at the Research Institute for High-Tech Computer Technologies of ITMO University as part of the project for implementing the resolutionGovernment of the Russian Federation, associated with the support of the development of cooperation of higher educational institutions and organizations.
The main tasks of MITP are the effective management of computing, information and software resources of distributed heterogeneous computing infrastructures, as well as the construction of corporate cloud infrastructures.
What can CLAVIRE
The CLAVIRE platform provides access to application packages that operate on remote computing resources. Currently, the CLAVIRE package database includes 65 software complexes in such areas as hydroaerodynamics, nanotechnology and quantum chemistry, analysis and modeling of social systems and transport infrastructure, hydrometeorology, shipbuilding and bioinformatics (this means that a number of complex multi-factorial solutions can already be solved on the CLAVIRE platform tasks in these areas - for example, to predict the likelihood of flood surges in St. Petersburg).
In addition, CLAVIRE provides multi-purpose calculations in SciLab, WEKA packages (demo packages available for public use through CLAVIRE interfaces can be found atlink ).
The CLAVIRE platform also allows you to design composite applications - complexes that interact to solve a common problem. A collection of composite applications for various subject areas was provided by the specialists of the Research Institute for High-Tech Computer Technologies of NRU ITMO and today contains several models.
Among them, it should be noted an application for calculating the distribution of temperature and mass velocities in the volume of the convection layer, a composite application for calculating the pitching characteristics of a vessel, and an application for solving the problem of calculating the dynamics of HIV spread.
The solution to the problem of calculating the distribution of temperature and mass velocities in the volume of the convection layer is based on modeling the dynamics of nanoparticles in a medium using the molecular dynamics method and numerically modeling the convection process using the finite element method (packages used: VMD, NAMD, Comsol, MDlogAnalyst, ComsolInitialise, Ruby interpreter)
How it works
The calculations themselves are performed on the capacities of Amazon EC2, OpenStack, using grid computing. Grid computing is a type of parallel computing that is performed using conventional computer systems connected to a network and combined into a “virtual supercomputer”.
The CLAVIRE system uses the GridNNS project (a grid system for the National Nanotechnology Network implemented in Russian supercomputer centers). The grid network of supercomputers is controlled by basic grid services located in the primary and backup control centers, the task of which is to coordinate the work of system resources and at the same time be a layer between users and the grid infrastructure.
Computing infrastructure. Source : CLAVIRE Cloud computing platform for high-performance computing / A.V. Bukhanovsky
To control the launch of tasks on multiprocessor installations, solutions such as PBS, Torque, Windows HPC, and Condor are used. Access to resources is via SSH or through web services, and Windows and Unix-like systems are used as operating systems. As for virtualization tools, CLAVIRE uses VirtualBox applications and VMware and KVM solutions.
The organization of the process of creating and executing a composite application under the control of MITP is reduced to the sequential formalization of sets of descriptions in terms of job flows (workflow, WF). At the top level of the application description is meta-WF. In it, individual blocks contain only instructions for performing design tasks.
Thus, MWF is a formal description of the user task in terms of the subject area, without indicating the conditions for its implementation. In addition to describing the actions and data necessary for the calculations, the user has the ability to set criteria and set restrictions by which resources and specific services will be selected.
Composite application in the form of WF. Source : Cloud Platform for High Performance Computing / A.V. Bukhanovsky, S.V. Kovalchuk
Designing a composite application with input data is a process of step-by-step refinement of MWF through the stages of abstract (AWF) and specific WF (CWF), up to the creation of specific scenarios for launching services in the cloud. At the first stage of designing a composite application, MWF is formed. The user is able to choose the classes of services that will be used when selecting services.
Schematic diagram of the creation and execution of a composite application. Source : CLAVIRE: Promising second-generation cloud computing technology / A.V. Bukhanovsky, V.N. Vasiliev, V.N. Vinogradov et al.
Next, WF is formed, in which specific implementations of computing services will be recorded. The next stage of design is scheduling and execution script in terms of CWF. For action blocks, services and nodes for execution are indicated, and for data blocks, the specific location of information.
The figure above shows a schematic diagram of the creation and execution of a composite application managed by MITP.
Where does it work
As an example of CLAVIRE's work, one can cite a system simulating the behavior of crowds in a stadium during the World Cup or other public events. The model takes into account the characteristics of the crowd, such as its social structure, and external factors, such as weather and political conditions. As a result, scientists see a clear picture of the behavior of people in given conditions:
Using the software system, you can recreate any territory and external conditions, and then place a certain group of people in this environment. This opens up tremendous opportunities for research. You can, for example, simulate an earthquake in a particular seismically dangerous region during a holiday, when thousands of people are on the streets of the city.
Sergei Ivanov, head of the international laboratory City Informatics, ITMO University, explained that for the basic operation of the models, ordinary outdoor surveillance cameras at different points in the territory are enough. This will allow you to adjust the model to the real situation, simply by highlighting groups of people from the video sequence and projecting them onto the model.
Field studyusing technology, it was recently conducted by ITMO University staff together with colleagues from other countries. The object of research was the religious festival "Kumbh Mela" in India. Within a month, researchers observed the pilgrimage of about 70 million people to holy places.
Scopes of MITP. Source : Cloud Platform for High Performance Computing / A.V. Bukhanovsky, S.V. Kovalchuk
The complex finds its application in medicine, when its task is to improve the health care system. Decision support systems become software assistants to doctors at all stages of work with patients. They allow you to summarize a huge body of data on the best medical practices and develop more effective methods of treatment and surgery.
PS If you are interested in the system, then a set of technical documentation, methods for using the CLAVIRE complex, program and operational documents, and initial skills for working with the platform can be obtained here . You can get to know CLAVIRE better and work with the system yourself here .