IoT architecture

Almost a year ago, I began to publish a series of articles on the architecture of IoT solutions. (Link to the first article habr.com/en/post/420173 ). And finally, the second article of the series goes to your court.


IoT projects look alike, and it’s true that they have common components. But at the same time, IoT projects have fundamental differences starting with the market request and the classes of tasks to be solved.

Obviously, developing each solution “from scratch” is extremely inefficient. Many components can be reused according to the principle of programming patterns.

In this article, we will analyze the various classes of IoT solutions and their respective implementations. As a result, recommendations will be given on the steps to implement the IoT architecture and we will try to highlight the general trends of IoT solutions.

We will also try to formulate the main problem areas, from the consideration of which, it is worth starting the construction of architectures for specific IoT solutions and possible problem areas of architectures.

IoT Applications

Today there is a division into three main geographical regions of application of IoT solutions:

Europe

IoT market in Europe is considering projects related to the conservation of natural resources. A typical example is day / night lighting, automation of space heating, water management, fire alarm.

Far East Region

In this market, the first priority for IoT solutions is security. Typical applications are cameras that detect unusual situations on the road, transport, and at home. The second major area is security and disaster prevention solutions such as seismic activity, hurricanes, typhoons, tsunamis.

North American Market

IoT projects aimed at optimizing business processes, such as optimizing transportation, efficient delivery of goods, as well as elements of smart homes and cities, are mainly represented on this market.

Of course, IoT solutions penetrate from one market to another and are not exclusive prerogatives. In each region, we can find all classes of IoT solutions. It is important for us to classify these classes of IoT products that exist on the global markets today. The following is a list of such classes of IoT solutions:

• Smart City. The main tasks are management and regulation of automobile traffic, night / day street lighting, hazard warning for pedestrians, determination of non-standard and dangerous situations in the city.
• Smart House. The main tasks are Security, an intelligent doorbell, TV and kitchen control, automatic irrigation and lighting systems, fire, gas, water leakage, and house temperature alarms.
• Weather and natural disasters. Meteorological information, seismic activity, fire control. Weather forecasting.
• Optimization of the use of resources in the home, city, country. Lighting, electricity, heating, optimization and forecasting the use of, for example, fuel in power plants.
• Optimization of transportation, delivery, storage and sorting. Companies such as DHL, FedEx use the solution to build optimal transport routes. Storage and sorting terminals at major airports.
• Factory monitoring and control, conveyor line management. Control robots. Sorting of goods, raw materials and testing of finished products.
• Sophisticated mechanisms, high-tech devices, such as modern cars, airplanes, etc. Automated control system, anti-theft protection, control of system units. Recognition of the face and body of the driver to prevent sleep, loss of attention. Prediction of maintenance and replacement of system components.

IoT architecture

General Topology of IoT Solutions

The figure below shows the layered architecture of IoT solutions. The IoT topology is different from a conventional layer model such as OSI. This is not a linear or more complex flow graph. Some components are optional and may not be available in a particular class of solutions. Two types of logic can be present - M2M (from car to car) and M2P (from car to man), as well as more special cases such as C2C (from car to car, usually in one LTE mobile communication cell).

The IoT solution has two physical locations - the first is the end (peripheral) devices, and the second is in the Backend data center on servers or in the cloud. At the same time, this is not a classic client-server application architecture, as we will see later.



Below we will analyze each level separately and compare its features with the actual classes of IoT solutions.

Physical Layer - physical layer

This level represents two types of operations - collecting information (Sensors) and performing mechanical work (Executive mechanisms).

Sensors can be divided into the following categories:

• Sensors:
  
  
  • Light: Photo diodes / transistors / resistors, PIR detectors
  • Sound: Microphones, ultrasonic sensors
  • Switches, in particular limit switches, registering the extreme points of mechanical movement. Measuring angle of rotation or speed of rotation.
  • Electromagnetic sensors measuring changes in physical characteristics, such as electrical capacitance, inductance, resistance.
• Complex or compound sensors. These include specialized sensors, such as gas, spectrum, etc., as well as a separate type of information collection device that is gaining increasing application - Video cameras.

In IoT solutions, physical elements have certain common requirements:

• As low a price as possible because of the high quantity in the IoT solution.
• Battery powered, which in turn requires low power consumption. Today's market demand is the operation of peripheral devices without maintenance from 1 to 10 years.
• Often located in remote and inaccessible places with minimal installation and maintenance costs.
• In the case of using video cameras, primary image processing with decision making based on artificial intelligence

Actuators of IoT solutions open the locks of entrance doors, actuate engines, selsyn, turn on / off the lights, heating, water, gas, etc. There are no major changes in the implementation of actuators. Therefore, this part of the IoT solution will not be covered in this article.

The table below shows the physical layer in various classes of IoT solutions:

We can summarize two problems for the requirements of the physical layer:

• Low power consumption. A high level of integration with the top layers is required.
• The use of video cameras. It also requires a high degree of integration with the upper layers and built-in AI / ML functions implemented in the peripheral device.

Edge Layer - Peripheral Computing Level

This level is usually connected to a single sensor or actuator. It provides minimal functionality for converting analog information to digital and / or vice versa. For connecting sensors, there are the same requirements for price and power consumption. Many manufacturers producing these types of devices do not have a single standard for data model, configuration, and operation, which creates individual integration problems.

To reduce power consumption, peripherals typically have four modes of operation:

• Sleeping mode
• Measuring and collecting information from sensors
• The mode of communication, transmission and receipt of information
• Installation and Connection Mode

The following is a block diagram of a peripheral device.

A peripheral device usually combines three levels: physical, peripheral computing, and communication. The main functionality of the peripheral computing level is the local ETL (Extract, Transformation and Load) - receiving, converting and saving information from sensors. This level is responsible not only for collecting information from the sensor, but also for bringing it to a standard form, filtering interference, preliminary analysis and local storage.

The table below shows the level of peripheral computing in various classes of IoT solutions:

So, the main requirements of the level of peripheral computing:
Low power consumption. This can be achieved with low-power hardware and Sleep / WakeUp algorithms. Often the presence of a local element of artificial intelligence.

Local Network Layer - peripheral communication layer

Data transfer is the most energy-intensive part of a peripheral device, as most peripheral devices are not connected to the mains and wired communications. In addition, peripheral devices can be located quite far from the Gateway (within a few kilometers). On the other hand, the amount of information transmitted is usually quite small. The following protocols are used at the peripheral communication level:

• ZigBee / Zwave
• Ble
• Lora
• Proprietary low band

Ad Hoc and Mesh are widely used today at this level to increase distance and reliability.

For configuration purposes, the NFC protocol can also be used. During the first installation and / or maintenance, a service engineer with a mobile application can connect to a peripheral device through a peripheral communication layer. Sometimes a Q-code printed on a peripheral device is also used for authentication.

The table below shows the level of peripheral communication in various classes of IoT solutions:

Gateway Layer - Gateway Level

There are several reasons for having a gateway layer in an IoT solution:

• If Backend receives raw information, it will increase its power and costs will be very high.
• Backend cannot guarantee real-time response for a large number of peripherals.
• Due to security restrictions, some information cannot be sent to Backend and cannot be continuously monitored by humans. Such information includes data from street surveillance cameras, medical information, etc.

The gateway should provide the following basic functionality:

• Implement a second level of ETL from their peripherals.
• Fix a critical situation and give out a local reaction, even without communication with BackEnd. This can be compared with the signals of a person’s heartbeat or lung breathing without the participation of the brain.
• Communicate with BackЕnd. Sends processed information from peripheral devices to the server and receives configuration data for peripheral devices.
• Save information on the status of peripheral devices, and data collected by them.

In some cases, AI / ML (artificial intelligence / machine learning) functionality must be present at the gateway level. The gateway device is mainly powered by the mains or has a large built-in battery, but some solutions also require low power consumption. In this situation, an additional problem arises - the synchronization protocol for communication with a peripheral device. One of them (gateway or peripheral device) should transmit the message “Ready for communication” more often than the other device is ready to communicate. The choice will depend on the total power consumption of each device and the required time without maintenance.

Today, we are increasing the number of applications with a source of information in the form of a video camera. In these specific solutions, Gateway and Edge can be integrated together. AI / ML functionality in such applications is becoming very popular. With the new machine intelligence accelerators for embedded systems, such a solution has become a reality.
Separately, it must be said about the gateways for smart home solutions. A gateway in this class of solutions is often combined with STB devices - television adapters or with a home security control unit. An open RDK-V platform already exists for the first integration. In the near future we should expect further integration of all three components - the gateway + STB + security into one device. It will also likely feature NAS (local file storage) and AI / ML services for machine video / audio recognition. Audio recognition devices such as Alexa are based on Cloud infrastructure, but primary recognition will likely be carried to the peripheral level.

The table below shows the gateway level in various classes of IoT solutions:

Wide Network Layer - External Link Layer

This layer separates the peripheral and BackEnd parts of the overall solution. The gateway is mainly connected to BackEnd using mobile wireless, such as 4G / 5G, but sometimes wired Internet access is used. The logical layer of external communication has a standardized protocol for IoT solutions called LvM2M. The LvM2M protocol was designed to access each peripheral device, but since many peripheral vendors do not support LvM2M interfaces, the gateway device can solve this problem and create a wrapper for communicating with peripheral devices.

The external communication layer also contains communication services and ISO models within itself. It includes DNS-based balancing and location services, the COAP transport protocol, DTLS encryption, and many other components that are beyond the scope of this article.

One important comment we have to make here. The LvM2M protocol uses the DTLS encryption protocol. The DTLS protocol is a protocol with security keys and a handshake session. It works on a point-to-point basis. To decrypt DTLS packets, we must use the same Back End instance that we had during the connection session. This poses a problem for the Load Balancer, which is part of the security level in our circuit. The load balancer, in turn, is necessary for automatic scaling at high system load. To avoid this limitation, DNS is used as a load balancer. Every N DNS query receives a new IP address for the security level instance.

Below is a table of the level of external communication in various classes of IoT solutions:

Security Layer - Security Level

This layer provides AAA (Authentication, Authorization and Accounting) and encryption / decryption functions along with other Internet-related services. All Cloud have their own security implementations, but functionally they are all built on the principle of roles and permissions. As noted in the paragraph above, this layer also serves as the terminator of the DTLS encrypted connection.

Endend Connectivity to Backend also has a security layer component.
The table below shows the level of security in various classes of IoT solutions:

Middleware Layer - level inside the server connection

This layer provides internal Cloud functionality for load balancing, message queuing and streaming. The components of this layer should be duplicated and scale automatically. The level is implemented mainly on the basis of microservices or PaaS from Cloud providers. This requirement stems from the paradigm of jumps and dips in data volume. Automatic scaling reduces the cost of backend implementation. The actual implementation of the service may be different, but the general principle remains one - to provide asynchronous message transfer with buffering and load balancing. In this way, the various Backend components can do their job independently and scale horizontally depending on the load.

The figure shows a schematic block diagram of intra-server communication patterns. Load Balancer is designed to distribute the load between different services. Queue - Queues provide intermediate buffering for implementing asynchronous operation of sequential services. Subscribers - recipients, subscribe to queues corresponding to their logic in order to receive messages sequentially after processing previous messages.

The table below shows the level of intra-server communication in various classes of IoT solutions:

Etl layer - level of data collection, processing and storage

The internal ETL level (extraction, conversion, and loading) is the third ETL operation. The first was in the peripheral device, the second in the gateway. Back End ETL collects data from all peripheral devices and gateways and is responsible for the following operations:

• Collection of information
• Bringing information to standard view
• Сохранение информации для дальнейшего использования
• Управление жизненным циклом информации включая архивирование и уничтожение
• Уведомление других сервисов о поступлении новых данных

The general implementation scheme for this layer is shown in the figure. The operation of collecting data (Extract) involves reading information from relevant queues. The transformation operation can be performed by specialized Cloud services, such as Lambda, or by computing means inside containers and just virtual machines. Each of the above methods has its positive and negative properties. For example, the Lambda service is convenient for almost complete automation, but has a significant creation time and therefore is not applicable if you need a quick reaction to the events that have appeared. Lambda is also poorly suited for continuous processing, since it is charged for time of use. The most commonly used service is containerized computing. They are conveniently scalable and easily ported to various BackEnds. The main objective of this operation is to bring the data to a convenient form for storing, sorting and searching. To do this, data is often combined from different messages and even queues.

Storage operations (Load) are intended for saving, sorting and subsequent information retrieval. Depending on the type of information and options for its use, various tools are used. If the data does not have a strict scheme (table columns), then it is stored in NoSQL databases. However, if the data can be systematized by a fixed schema, then SQL database types are used. The latter, in turn, have 2 types - OLTP (Online Transactional Processing) and OLAP (Online Analytic Processing). As the name implies, the first type is more suitable for the ETL process itself - writing new values ​​to the database, while the second is more convenient for searching and analyzing data. Therefore, often after downloading to the OLTP database, in the background, the data is copied to OLAP. There are situations when data is not convenient or not possible to store in databases, For example, as a record, This data is recorded in a Bucket, and the metadata of the records is stored in databases. To reduce storage costs, obsolete data is archived or deleted. And the last component of this level is the internal notification of the availability of new stored data for presentation to customers and for analysis services.

The table below shows the level of collection, processing and storage in various classes of IoT solutions:

Big Data and Analytic Layer - analytics layer

Depends on the specific IoT application. Big data and analytic level will extract situational information from the entire set of peripheral devices. This part is less standardized, because it is very different from one application to another due to various problem solving. AI / ML algorithms are also widely used in this layer.
A separate category is the prediction of future events, such as necessary parts in a warehouse, consumption of future resources, weather, etc.

The table below shows the level of analytics in various classes of IoT solutions:

Notification layer — уровень уведомления

There may be several components at this level, but they all have a subscription notification algorithm. The client application subscribes to the necessary events and, when this happens, receives an information signal - a notification. These are mainly email applications and mobile clients, fewer phone calls (used for emergency alerts). The mobile application is forced to go into sleep mode for power consumption, but the iOS and Android OS have a notification mechanism indicating the arrival of new data.

The table below shows the notification level in various classes of IoT solutions:

Presentation Layer — уровень представления

An IoT application can have two streams: M2M (from machine to machine) and M2P (from machine to person). The presentation layer associated with the M2M stream, where the Back End processes information and provides it to a client or support engineer. Today there is no standardized UI / UX representation for this level, but I hope that it appears in the near future.

The presentation layer is also responsible for maintaining, configuring, and changing the state of the system, including peripherals and gateways. It also includes commands for controlling the actuators of peripheral devices.

The table below shows the presentation level in various classes of IoT solutions:

Configuration Layer - configuration level

This level applies to both streams - M2M and M2P and works as a storage for three types of statuses of peripheral devices:

• The current status of the peripheral device
• Новое состояние периферийного устройства, которое будет загружено.
• Промежуточный статус периферийного устройства — указывает на процесс обновления от старых состояний к новым. Часто этот статус отсутствует.

A peripheral device and even a gateway can only have a short connection time to the Backend. We discussed this earlier. Any change in status from a client or system is stored at this level, and is sent to the gateway or peripheral device during the communication time.

In order for this logic to work, the following communication process is usually implemented:



If the gateway is present in the information transfer scheme, then most of the information from peripheral devices is sent to the server part in the form of data packets collected from several peripheral devices.

The table below shows the configuration level for the various classes of IoT solutions:

Results and how to build an architecture of IoT solutions.

To summarize the above. The following development trends are observed in IoT solutions:

• Sensors are divided into 2 groups:
  
  • Simple, cheap, with the lowest possible energy consumption. Low speed and high range information transfer. These are actually one-time devices that are not serviceable.
  • Based on camcorder. The device is integrated with a peripheral computer. It has built-in mechanisms for pattern recognition and basic decision making.
• ETL процессы происходят на нескольких уровнях — периферийных устройств, шлюзов и в backend. Пока нет единого подхода, что должно делаться на каждом уровне, но идеология такова — все что можно обработать должно быть обработано на возможно более низком уровне.
• Основным, универсальным средством передачи информации является беспроводной интернет. Но фактические протоколы отличаются на разных уровнях. Логический уровень связи — протокол LvM2M.
• Backend в большей степени Cloud. Большинство решений на сегодняшний день используют AWS.
• В качестве устройства отображения фактической информации используется мобильное приложение, а аналитическая информации представляется WEB приложениями. Экстренное оповещение через мобильный звонок с предустановленным голосовым сообщением. Электронной почтой получаются в основном отчеты.

Where to start building an IoT architectural solution? There is no single approach to the answer to this question. And here I will give my personal opinion:

1. Define the data model that we can get from the gateway, i.e. transferred to Backend.
2. Check which peripheral devices can collect data and how it should be processed to bring to the model transmitted by the gateway.
3. Check the requirements for peripheral devices - distances, amount of information, power consumption, etc.
4. Choose the appropriate peripheral computing device, its location in relation to the sensors, the protocol of their work.
5. Solve the architecture of the Cloud part, including:
   
   
   • Security
   • Load distribution
   • Cloud Asynchronous Data Transfer
   • Elements storage, form and data life cycle
   • Build a graph of system information transfer
   • Build analytical models, AI / ML component
   • Develop types and content of notifications
   • Set up redundancy and auto scalability of services
   • Estimate cost and optimize
6. UI / UX Design for Mobile Clients
7. Build a peripheral device data transfer feedback

I hope this article will be useful, at least for the first introduction to IoT projects. In the future, I will try to give specific implementations of IoT solutions.