How we served the IT infrastructure of “Luzhniki” during the World Cup
I had the opportunity to participate in a project to create the engineering and IT infrastructure of the Luzhniki Grand Sports Arena. The work lasted for several years and ended with the support of the solutions implemented by us during the FIFA World Cup. Under the cut - how we implemented and maintained HD Wi-Fi on the country's largest sports ground, what problems we faced and how Raspberry Pi minicomputers helped us.
A source
The main news of the day: At the BSA Luzhniki there will be two Wi-Fi networks for fans, not just one.Now Wi-Fi hotspots for fans are installed only in sky boxes and under the stands. In 2014, when the reconstruction began, the customer decided not to build Wi-Fi in the stands of the stadium, since this message was present in the FIFA requirements in the form of a modest wish. The customer now has an official letter signed by FIFA CIO Dick Wiles (Dick Wiles). The essence of the letter is that now HD Wi-Fi in the stadium stands is not a soft request, but a strict requirement. It is not very clear what to do with it now. The access points and Wi-Fi HPE controllers purchased in 2014 and installed according to the project went into such an end-of-sale that there is no possibility to expand the network on existing controllers. We will build another Wi-Fi network at Luzhniki. Ideally, the solution should go through trial operation on March 23, 2018, at the test match Russia-Brazil. At 100% everything should work on June 14, at the opening ceremony of the FIFA World Cup.
Since this is not the first project of LANIT-Integration for the construction of HD Wi-Fi in stadiums (you can read about the first project here ), the main points that you should pay attention to are already clear.
WiFi antenna at the stadium. Please note that it is located behind the glass, which somewhat degrades the signal quality, but physically protects the antenna from the fans.
The customer made the final decision that the second Wi-Fi network at the stadium will be built on Cisco Systems equipment. According to the most preliminary calculations, about 500 access points should be hung in the stands of the stadium. Cisco engineers suggest doing a project on AIR-CT8510-500-K9 controllers and on AIR-CAP3702E points with AIR-ANT2513P4M-N = and AIR-ANT2566D4M-R = antennas. If we order equipment today, it will come at least in a month and a half, that is, on February 1. And on March 23 - a test match. The timing is extremely tight, so the controllers decided to order immediately. To monitor the IS incidents, it was decided to add to the solution and also immediately order a pair of NGFW based on the ME of Fortinet in the gap between our network and the carrier network. Ordering access points and radio antennas is still dumb, for we vaguely imagine how many points you need and where to hang them.
A source
It turns out that during the World Cup there will not even be two Wi-Fi networks, but three. Another independent wireless network (on separate controllers) will be deployed by FIFA for its own needs and the needs of media professionals. Since the FIFA network partially affects the stadium area, the existing Wi-Fi HPE access points in some rooms should be temporarily disabled, and even better, dismantled. Otherwise it does not affect the course of our project.
We decided on where we will conceptually hang access points in the stands of the stadium. Four options were considered: running walkways, under seats, along the sectors perimeter (on the railing, portals for access to the tribune) and under the stands, in the hope that the radio signal would penetrate the floor slab (there is an exotic way, but there are projects in the world) .
The first option was to drop the option to punch the slab: the attenuation of the signal in the 2.4 GHz band was measured with the through passage of the slab. It was -80dBm, and that buried the solution.
Now it was the turn to abandon the placement under the seats, because the variant meant significantly more access points (about 800) and required to practically riddle reinforced concrete slabs with a thickness of 1 m with hundreds of technological holes for cable supply, which do not understand how to waterproof.
The installation on the roof looks most tempting: laying the cable along the walkway is much easier, especially since there are even optics and climatic telecommunication cabinets. But even this is not destiny: the average distance from the running walkways of the roof in a straight line to the stands of the Luzhniki Stadium is about 40 m, with the design standard a maximum of 20 m for HD Wi-Fi.
It remains the only option - to scatter points around the perimeter of the stadium sectors. This means the inevitable unevenness of radio coverage plus the elaboration of solutions to protect against equipment encroachments by guests. There are simply no other ways to fulfill the requirements of FIFA.
We divided the stands of the stadium into typical sectors and conducted a radio survey of one typical sector. For the survey, we asked Cisco Systems for a test point AIR-CAP3702E with antennas that the vendor offered. According to the results of the survey, it was possible to outline the ways of fastening the equipment. To prevent fans from reaching the access points, special mounting brackets are needed. We'll have to make them to order, and several types of brackets are expected, depending on the typical installation locations of access points. We also need mounting boxes, because The access points offered by the vendor are not vandal resistant, and Cisco's vandal resistant access points for HD Wi-Fi are much more expensive and will not appear on the site for the Russia-Brazil match.
The designers received the results of the radio examination of one sector of the stadium and, using this data, drew the task for the supply of cables to the SCS in all sectors. Determined with the required number of access points - 430 pieces. This made it possible to start purchasing points, issue a task to the manufacturer of brackets and mounting boxes, as well as estimate which cross-over ones need to install additional access switches in order to be able to connect our access points. It is already clear that the project requires 12 different types of mounting brackets. The purchase of the brackets will have to be limited to the pilot batch so far, since the customer must first see the solution in kind and make sure that the access points do not interfere with anyone. The specification for access switches, antennas and Wi-Fi points was sent to the order in full. If the ordered equipment comes at the end of February,
Fragment of the assembly drawing of one of the brackets used at the stadium
It is cold at the stadium, but there are no severe frosts, and this allows our installers to pull cables to access points in two shifts. The first pilot samples of the mounting brackets arrived. Since the access points have not yet arrived, they decided to do a pilot installation using the equipment previously taken from Cisco Systems for the radio survey.
In the meantime, the engineers are doing their best to work out the solution for SMS authentication of subscribers. A separate subsystem should be responsible for this. There are several software manufacturers and telecom operators on the market from which it is possible to buy this subsystem, either as software or as a service. We settled on a product called WNAM from LLC Netams, which worked with us at the Otkrytie Arena stadium. This is how it should look in general terms.
General system architecture
WNAM communicates with Wi-Fi controllers using the RADIUS protocol. When the subscriber tries to connect to the SSID, the controller asks WNAM if there is a client with the MAC address from which the connection request came in the registered user database. If there is - the client gets access to the Internet and optionally redirects to the starting website (in our case, this is the FIFA portal welcome2018.com). If it does not exist, the controller redirects the user to the authentication portal, and so far, except for this portal, it does not allow it. The portal is a web page that spins on the WNAM authorization server. The design of the authentication portal for each project is unique and is developed separately (in our particular case, it was sent by the FIFA Organizing Committee). During FM, the authentication start page should look like this:
As soon as the user enters his phone number, WNAM generates an authorization code and sends it as SMS via the kannel utility via the SMPP protocol to the SMS operator, and that via the appropriate cellular operator to the final subscriber.
In our case, in order to communicate with the operator, we need an IPSEC tunnel and, of course, an agreement under which this SMS operator will work with us.
For sending each SMS-message operator will be charged. The one who signs the contract, in this case the owner of the Wi-Fi infrastructure, must pay. (Actually, you can tell the subscriber a one-time code on the web portal, and then ask him to send this code to the number of the SMS operator. In this case, the subscriber will pay for sending the SMS, but we did not go that way).
Upon successful delivery of the SMS message, the cellular operator generates in the opposite direction a delivery report with confirmation of the fact of receiving the SMS. This message is transmitted by the SMS operator to WNAM, so we know whether the subscriber received a one-time password or not.
The user is prompted to enter a password on the portal. If the password is correct, WNAM instructs the controller to authorize the subscriber and give him access to the Internet.
Wi-Fi controllers, access switches and firewalls arrived at the site. The equipment was assembled and started up and commissioning. Colleagues from Netams started setting up WNAM. In the meantime, the customer was called from the relevant departments and was reminded of the high responsibility that rests with all of us in working out potential information security incidents at the World Cup matches.
To save time, we actively negotiate with the customer the pilot assembly units informally. We explain that the antennas and access points have already been purchased and are going to the site, so the maximum that we can change under extremely tight deadlines is to move points a little or slightly fix brackets. Otherwise, the project can be minimized. The customer goes forward and makes minimal adjustments to the pilot solutions. Thus, we slowly agree on the order of design batches of typical mounting brackets.
The organizing committee sent updated requirements for the work of the Wi-Fi network for fans. The main changes concern the statistical reports that the Organizing Committee will require from us at every World Championship match. Some requirements are very specific. In addition to the obvious things such as communication channel load schedules and the number of subscribers, it is required to calculate the ratio of users in the 2.4 and 5 GHz bands, as well as data on how subscribers go through various authentication stages, and even broken down by country from which these subscribers came . The FIFA Organizing Committee wanted to see statistics not only in the form of interactive graphs, but also in the form of tables with the ability to automatically export data to excel and pdf, and the option of setting a time period for which data is needed should be available before export. Such requirements suggested a separate “umbrella” system for collecting and analyzing statistics, which will summarize data from various service components, since information on the utilization of communication service provider channels, for example, must be taken from the firewall, authentication statistics — from WNAM logs, and data on the number of subscribers in the 2.4 and 5 GHz bands - only on controllers and nowhere else. So in our project appeared heavy artillery in the form of a system for collecting and analyzing statistics Splunk Enterprise. 4 and 5 GHz - only on controllers and nowhere else. So in our project appeared heavy artillery in the form of a system for collecting and analyzing statistics Splunk Enterprise. 4 and 5 GHz - only on controllers and nowhere else. So in our project appeared heavy artillery in the form of a system for collecting and analyzing statistics Splunk Enterprise.
A source
But what to do with complaints about the poor performance of Wi-Fi? How to evaluate the work of the service, if you control not all of its components, there are too many "floating" problems, and the descriptions of incidents received from subscribers only take time? Why not scatter around the stadium some devices that will behave on the Wi-Fi network in the same way as our users, and at the same time send us detailed logs about how things are in reality? These should be devices that could connect to the Wi-Fi network themselves, authenticate to WNAM, work on the Internet, disconnect and do it all over the script again and again, sending us detailed logs at each step. Of course, all this should still be inexpensive, as the project budget is almost exhausted. It will be necessary to discuss this topic with the engineers, it may be possible to come up with something.
We discussed with engineers the idea of proactive monitoring of Wi-Fi by simulating user activity. Remember about the Raspberry Pi. In theory, it can be endowed with the functionality of a mobile device, if equipped with a GSM modem with a SIM card, which will receive SMS with a one-time authentication code. It is possible to feed a product on PoE, through a PoE-extractor.
Appearance Raspberry Pi 3 model B without body
Engineers issued a project of a proactive monitoring device based on the Raspberry Pi 3 Model B. It is supposed to connect the D-Link DWA-171 / RU / A1A Wi-Fi adapter to it, since the internal Wi-Fi adapter does not support networks in the range 5GHz (sorry that the B + version, where there is an integrated adapter for both bands, will be released a month later), as well as a Huawei E3372h-153USB 3G / GSM modem with a SIM card. The power supply of the minicomputer can be carried out via the Upvel UP-102S intermediate PoE extractor. All this will be packed in a plastic case.
On the minicomputer running the Raspbian OS, the Firefox Internet browser and the Splunk forwarder component are installed, which should send logs to Splunk Enterprise. The microcomputer activity scenario is defined by a separate python script. Logs are sent via wired Ethernet, not via Wi-Fi, which is another advantage of the solution.
The operation of the verification script is based on interaction with the browser and performing actions similar to those performed by the user in the process of working with Wi-Fi (loading the authentication page, entering the number on which the SIM card is registered, entering the received code via SMS, several speed tests on the Internet). For this ideally suited library Selenium, created for the language Python.
Here is a product. In the photo - the case with the cover removed.
Customer liked the idea with Raspberry. He agreed to include 50 minicomputers in the solution and purchase three cellular operators for them SIM cards. Now, if during the match someone calls and starts complaining about the poor performance of Wi-Fi, we will have 50 reference devices, by which you can see the overall picture of the service at the stadium.
The first batch of mounting brackets arrived. All access points and antennas are already at the facility, the SCS cables are stretched, and there are not enough brackets. We mount what is. We agreed with the customer that by the Russia-Brazil test match, access points will be mounted on typical sectors in order to be able to verify the correctness of the decisions made for the placement. It is a pity that before the opening ceremony, we definitely will not have time to test the Raspberry.
We were invited to the next meeting on information security. At the meeting, we asked what IS risks during the Championship would be the most significant. It turned out that the most unpleasant thing that could happen was the broadcasting of extremist content to the video board of the stadium. We asked what would happen if someone thought of it during the match to display pornography on the scoreboard. We were told that this, of course, is also unpleasant, but there are worse things.
Today we have completed setting up traffic mirroring on the SORM-2, SORM-3 and SOPKA server interfaces, which are installed directly in the stadium data centers. I had to tinker a bit with redirecting the logs from WNAM and the firewall to SORM3, so that if there was any where to get the private IP address, MAC address and SIM number of anyone authorized in our network.
Today at the stadium there is a friendly match between the national teams of Russia and Brazil, so the installation of access points is suspended. For the test match, we managed to hang up access points only on the lower tier of the stands of the stadium (this is 50% of the total) plus Wi-Fi in the sub-bed space, which was already working. The main task for today is to make sure that our typical solution for the arrangement of access points will work in conditions of a large cluster of subscribers, and of course, the general maintenance of the IT infrastructure is also ours.
Our team is located on the ground floor of the eastern stands of the stadium in a room with white walls and no windows. There are only IT people, and we are about fifteen people. The people are sitting and staring intently at laptops. Someone initially lacks chairs. Still indoors it is stuffy and a lot of white light. All of this suggests thoughts of Orwell’s cameras in the Ministry of Love, but there’s no time to think about it.
Each of the engineers is responsible for the operation of a separate IT system: data storage, servers, virtualization, AD DS, Wi-Fi ... We have a separate table with the project manager and a large monitor that displays Splunk Enterprise monitoring systems. Splunk Enterprise collects logs from all IT systems in the stadium, for which we are responsible. It also shows the statistics of high-density Wi-Fi.
The statistics is very detailed, there is even a percentage of user distribution over the 2.4 and 5 GHz bands (they thought for a long time how to implement, and finally decided to periodically use Splunk via the API to access Wi-Fi controllers from Cisco and HPE through intermediaries in the face of the HP IMC monitoring system to which HP point controllers are connected, and Cisco Prime to which Cisco controllers are connected).
Since there are no windows in the room, we have no idea what is happening on the field. Therefore, in the corner of the room there is a TV that broadcasts the live broadcast of the First Channel. On this TV, we with the project manager will watch all the matches of the 2018 World Cup at the Luzhniki Stadium.
From the room where we sit, there is a short passage to one of the entrance turnstile groups. That is, you can go out and see how people pass through the turnstiles to the stadium. From this, in general, the everyday process is impossible to break away, because the turnstiles are synchronized with the ticket server on the network, for which you answer with your head. It is terrible to imagine what will begin if the turnstiles stand.
People are pulling up to the stadium. We see how they pass through the turnstiles, and we observe in Splunk dashboards how the number of connected users is growing. It grows evenly until the starting whistle, then falls, in the break between the first and second half, another peak, then a recession again, and finally a small final surge after the end of the match. No incidents have been recorded. The decision on the arrangement of access points in the stands of the stadium was recognized as workable.
As you know, in order to identify a person by phone number, you need to make a request to the telecom operator to obtain passport data of the person to whom the number is registered. If you have registered a SIM-card in Iran, Mexico, Morocco, it’s not a fact that you will be able to identify you promptly through a request to the cellular operator.
Therefore, the Organizing Committee sent the updated requirements for SMS authentication. If at the World Championships someone tries to register on a Wi-Fi network using a foreign number, they will be asked to enter a FAN-ID number or fan passport. The implementation of this requirement requires separate dances with a tambourine at the WNAM level in terms of integrating the SMS authentication system with the FAN-ID databases. Especially since the FAN ID portal with which to integrate is still under active development.
Added an authentication portal page for foreign SIM-cards to WNAM. Since the FAN ID database is not yet operational, the FAN ID authentication does not work yet.
Finally completed the installation and commissioning of all access points. The system is ready for operation. Unfortunately, we have not managed to mount the Raspberry yet, but there’s nothing really terrible about it. We will carry out these works until June 14.
Today we were offered to check how the Wi-Fi will work if the stadium launches electronic warfare against unmanned aerial vehicles. During the test, Wi-Fi revealed that the signal level has become much worse. We were left to work with electronic warfare equipment, and this resolved the issue. Thanks for the constructive attitude.
I am standing in the corner of the Luzhniki conference hall, and in the conference hall, preparations are underway for the meeting before the opening ceremony of the World Championship. Next to me is a box of beer, on which lies a piece of paper that says “Rider R. Williams. Do not take". The opening ceremony will begin in a few hours.
We have just finished the last thing we had to do at the stadium before the World Cup - we mounted and connected a TV panel on the presidential VIP-platform, which will display match statistics and repetitions of interesting moments. They could not resist and sat in the chairs in which Vladimir Putin, Crown Prince of Saudi Arabia Mohammed bin Salman and FIFA President Gianni Infantino would soon be sitting.
Source
Each of us has accreditation to the stadium. To get it, each filled out a questionnaire and fingerprinted. Our accreditation gives the right to move freely around the stadium, but in the period three hours before the start of the match, during the match itself, as well as during the rehearsals of the events and training of the teams, we are not allowed to enter the field and stands. Therefore, three hours before the match, we go down to our headquarters and turn on the TV. Fans are slowly catching up. The number of subscribers who connect to Wi-Fi is gradually increasing.
The stadium's IT infrastructure completed the match without incident. However, the low percentage of users who logged into the SMS authentication portal and successfully passed it confuse - 30% with an expected 50%. We assumed that everything was due to the five goals scored by the Russian national team at the gates of the national team of Saudi Arabia, but the Organizing Committee does not agree with us. Let's see what will happen in the next matches.
Source . The girl decided to take pictures on the background of antennas HD WI-FI
We analyzed the WNAM logs and found out that during yesterday's match a lot of people for some reason, when registering on the WiFi HD portal, incorrectly entered their FAN ID for some reason. If you look at the passport of the fan, then there are three numbers, similar to those that require authentication portal page. Assumed that the fans did not know what exactly the number you want to enter. As a result, together with the FIFA Organizing Committee, we came up with a couple of ideas that should improve the quality of the Wi-Fi service at the stadium:
In the midst of the match from the stewards comes a complaint about the problems with access to Wi-Fi in the VIP-zone. We are escorted to the sky box. In the sky box, a table is set and a crowd of Mexicans hang out. In the corner - the bar. It is downright bursting with a variety of alcohol. Near the bar stand is a hefty Mexican. He has beer and a phone in his hands. It's hard for him to stand. He was the one who complained about problems with Wi-Fi.
We communicate in English. We explain that in this field the Mexican must enter his mobile number:
- Tell us your phone number. Do you remember your phone number?
Mexican staggers. At first, he doesn’t remember his phone number at all, but on the third attempt he still remembers it. We continue to explain:
- Now you must enter your FAN ID number. Show your fan passport, please.
“Listen,” he says , “I just want to post a selfie on Instagram.” In my opinion, for self it is all too difficult. So I’ll just watch football and get by without selfie and without you. You are very hospitable, goodbye.
Internet access for a Mexican, we still organized. And at the end of the match I was sent a video of how the fans poured beer on the steward. It is not easy to remember your phone number when the magic of Mundial is raging around you.
A source
Every time after the match, we hold an internal meeting at which we discuss the outcome of the event, whether it is possible to improve the quality of the infrastructure and how to do it. This time the meeting was devoted to the decision on the Raspberry Pi, which during the match Russia-Mexico worked in full force, but surprised us with strange statistics. In the morning, when there was no one at the stadium, Raspberry informed us that Wi-Fi was working fine. However, as soon as the fans began to take their places, all minicomputers began to inform us about the complete lack of service. At the same time, Splunk and the WNAM reporting subsystem reported that the connections were going fine. It turns out that the data contradict each other.
During the step-by-step debugging of the Raspberry script, it turned out that it needs to be substantially refined, since in its current form it gives a huge amount of false errors. It was decided to release a new version of the script, which will contain 15 additional checks. Replaced Firefox's fairly heavy browser with PhantomJS, which ultimately made it possible to double the speed of checks. Another possibility of the system was to display on the pages of statistics in Splunk screenshots of the authentication page for visual assessment of possible problems.
To quickly fill the new versions of the script on 50 devices, we decided to install the Ansible orchestrator.
An analysis of the script also revealed completely unobvious things: for example, there was a moment when all the Raspberry started swearing at the wrong one-time passwords. It turned out that all microcomputers, in addition to the codes, received other SMS messages, for example, spam or messages from the Emergencies Ministry about the deterioration of the weather. In the scan cycle, such SMS messages were buffered before messages generated by WNAM, and, accordingly, were perceived by the script as a one-time password.
“What the hell is Morocco doing about this?” Exclaimed our engineer in charge of monitoring WNAM in a rush when he saw a growing queue of sending outgoing SMS messages without a delivery report. We in the logs had statistics on sending SMS-messages for all operators who have codes in open databases, and we could see which countries of the SMS arrive normally, and which ones - with a delay. The Kingdom of Morocco has a strong outsider position in our statistics. At the beginning of the second half, it was decided to redirect the newsletter to another SMS operator, and this somewhat improved the situation.
The source
On TV, the live broadcast of the match from the stadium was usual. We need to view the broadcast in order to feel the dynamics of the activity of Wi-Fi clients: during the game it falls, and during breaks it grows on the contrary.
Compared with the match “Portugal-Morocco”, the accompaniment turned out to be quieter, since Denmark and France - European countries and statistics by SMS were expectedly better for them than for Morocco. Therefore, we dedicated the match to debugging Rasberry scripts and became so enthusiastic about this process that only at the end of the match we understood: all this time a completely different match was broadcast on TV, not from the Luzhniki stadium, but from the Fisht stadium.
In the subway after the match, a peasant bore me with questions:
- Listen, and who just played at the Luzhniki Stadium?
I hardly remember the names of the teams:
- Denmark and France. Yes, France.
- What is the score?
I don’t remember the score at all, because I didn’t watch the match, and I reported it. When the man realized that I was going from the stadium and did not know the bill, he looked at me with unconcealed amazement.
In the middle of the second half on the firewall, a suspicious network activity of one of the connected subscribers was discovered. The firewall blocked from its IP address a large number of connections on non-standard ports (which were closed by the firewall policy). Inquiries went to Russia, the CIS countries and Europe. Since this was an information security incident, security guards enthusiastically got down to business.
The WNAM logs have calculated the phone number to which the subscriber is registered. Called to the specified number. Introduced themselves to the Luzhniki service. They told some peasant that we were seeing suspicious network activity from the phone and therefore we would like to clarify if he is a hacker. The man at the other end of the line told us that he understood absolutely nothing. We said that in this case, if he does not object, we will block his access to the network. The man did not mind. There was some impression that he even supported this initiative. We blocked access and transferred the incident logs to the customer.
In the middle of the first half, a Luzhniki operation service representative came to our headquarters. He brought with him a spectator whom we called during the last match. It turned out that the man showed an active position, contacted the Luzhniki exploitation service and asked to arrange a meeting. He brought the phone with him and stated that he had not made any hacking attempts, so he was somewhat alarmed and would like to know what his complaints were.
He turned out to be an inexpensive Android smartphone from Samsung, a quick inspection of which revealed the torrent client installed on it, as well as many other software of dubious origin, which probably generated suspicious activity. We recommended a person to scan the phone with an antivirus, remove excess software and refrain from using the torrent client.
What non-obvious features were revealed in the solution during the commissioning and trial operation of HD WiFi on the BSA Luzhniki:
November 13, 2017
The main news of the day: At the BSA Luzhniki there will be two Wi-Fi networks for fans, not just one.Now Wi-Fi hotspots for fans are installed only in sky boxes and under the stands. In 2014, when the reconstruction began, the customer decided not to build Wi-Fi in the stands of the stadium, since this message was present in the FIFA requirements in the form of a modest wish. The customer now has an official letter signed by FIFA CIO Dick Wiles (Dick Wiles). The essence of the letter is that now HD Wi-Fi in the stadium stands is not a soft request, but a strict requirement. It is not very clear what to do with it now. The access points and Wi-Fi HPE controllers purchased in 2014 and installed according to the project went into such an end-of-sale that there is no possibility to expand the network on existing controllers. We will build another Wi-Fi network at Luzhniki. Ideally, the solution should go through trial operation on March 23, 2018, at the test match Russia-Brazil. At 100% everything should work on June 14, at the opening ceremony of the FIFA World Cup.
November 15, 2017
Since this is not the first project of LANIT-Integration for the construction of HD Wi-Fi in stadiums (you can read about the first project here ), the main points that you should pay attention to are already clear.
- All clients must authenticate before accessing the Internet.Если речь идет о гостевом доступе в Интернет в общественном месте, это требование российского законодательства. Наиболее понятный способ решить вопрос – предложить абоненту сообщить свой номер мобильного телефона в обмен на получение одноразового пароля по SMS.
- Access points will not stand where it is optimal, but where allowed.Точка доступа может висеть на стадионе только при одновременном соблюдении трех условий: а) не заслоняет обзор; б) руками не дотянуться; и в) можно подвести «витую пару». Как только план расстановки точек доступа попадет на согласование людям, ответственным за общественный порядок, в случае несоблюдения хотя бы одного из условий точку демонтируют, а любые доводы типа радиообследования, рекомендаций вендоров и т.д. будут трактоваться как отговорки в пользу бедных.
- There are always complaints about the work of Wi-Fi, which is not clear what to do.«Что-то я слишком долго скачиваю файл через WhatsApp». «Вчера моя девушка была на стадионе и не смогла подключиться». Крайне редко претензия содержит сведения, хоть как-то помогающие в работе. В разгаре матча совершенно без толку уточнять у человека MAC-адрес телефона, просить прислать скриншоты и т.д. Если мы не хотим бегать по стадиону в бесконечных попытках проверять качество работы HD Wi-Fi, надо придумать, что делать с подобными обращениями.
- It is difficult to objectively understand how well HD Wi-Fi works in a stadium.Абонент может быть недоволен качеством Wi-Fi из-за кучи «плавающих» вещей, которые даже неподконтрольны тому, кто занимается обслуживанием ИТ-инфраструктуры объекта. Может не прийти SMS-код из-за глюков телефона, загруженности сотовой сети или сбоя на стороне SMS-оператора. Возможны задержки открытия web-страниц из-за проблем на стороне операторов связи, или хостинг-провайдеров. Человек банально не поймет, как подключиться… Качества сигнала и отсутствия помех в беспроводной среде передачи данных по определению никто не гарантирует.
December 15, 2017
The customer made the final decision that the second Wi-Fi network at the stadium will be built on Cisco Systems equipment. According to the most preliminary calculations, about 500 access points should be hung in the stands of the stadium. Cisco engineers suggest doing a project on AIR-CT8510-500-K9 controllers and on AIR-CAP3702E points with AIR-ANT2513P4M-N = and AIR-ANT2566D4M-R = antennas. If we order equipment today, it will come at least in a month and a half, that is, on February 1. And on March 23 - a test match. The timing is extremely tight, so the controllers decided to order immediately. To monitor the IS incidents, it was decided to add to the solution and also immediately order a pair of NGFW based on the ME of Fortinet in the gap between our network and the carrier network. Ordering access points and radio antennas is still dumb, for we vaguely imagine how many points you need and where to hang them.
December 18, 2017
It turns out that during the World Cup there will not even be two Wi-Fi networks, but three. Another independent wireless network (on separate controllers) will be deployed by FIFA for its own needs and the needs of media professionals. Since the FIFA network partially affects the stadium area, the existing Wi-Fi HPE access points in some rooms should be temporarily disabled, and even better, dismantled. Otherwise it does not affect the course of our project.
December 20, 2017
We decided on where we will conceptually hang access points in the stands of the stadium. Four options were considered: running walkways, under seats, along the sectors perimeter (on the railing, portals for access to the tribune) and under the stands, in the hope that the radio signal would penetrate the floor slab (there is an exotic way, but there are projects in the world) .
The first option was to drop the option to punch the slab: the attenuation of the signal in the 2.4 GHz band was measured with the through passage of the slab. It was -80dBm, and that buried the solution.
Now it was the turn to abandon the placement under the seats, because the variant meant significantly more access points (about 800) and required to practically riddle reinforced concrete slabs with a thickness of 1 m with hundreds of technological holes for cable supply, which do not understand how to waterproof.
The installation on the roof looks most tempting: laying the cable along the walkway is much easier, especially since there are even optics and climatic telecommunication cabinets. But even this is not destiny: the average distance from the running walkways of the roof in a straight line to the stands of the Luzhniki Stadium is about 40 m, with the design standard a maximum of 20 m for HD Wi-Fi.
It remains the only option - to scatter points around the perimeter of the stadium sectors. This means the inevitable unevenness of radio coverage plus the elaboration of solutions to protect against equipment encroachments by guests. There are simply no other ways to fulfill the requirements of FIFA.
December 25, 2017
We divided the stands of the stadium into typical sectors and conducted a radio survey of one typical sector. For the survey, we asked Cisco Systems for a test point AIR-CAP3702E with antennas that the vendor offered. According to the results of the survey, it was possible to outline the ways of fastening the equipment. To prevent fans from reaching the access points, special mounting brackets are needed. We'll have to make them to order, and several types of brackets are expected, depending on the typical installation locations of access points. We also need mounting boxes, because The access points offered by the vendor are not vandal resistant, and Cisco's vandal resistant access points for HD Wi-Fi are much more expensive and will not appear on the site for the Russia-Brazil match.
December 28, 2017
The designers received the results of the radio examination of one sector of the stadium and, using this data, drew the task for the supply of cables to the SCS in all sectors. Determined with the required number of access points - 430 pieces. This made it possible to start purchasing points, issue a task to the manufacturer of brackets and mounting boxes, as well as estimate which cross-over ones need to install additional access switches in order to be able to connect our access points. It is already clear that the project requires 12 different types of mounting brackets. The purchase of the brackets will have to be limited to the pilot batch so far, since the customer must first see the solution in kind and make sure that the access points do not interfere with anyone. The specification for access switches, antennas and Wi-Fi points was sent to the order in full. If the ordered equipment comes at the end of February,
January 15, 2018
It is cold at the stadium, but there are no severe frosts, and this allows our installers to pull cables to access points in two shifts. The first pilot samples of the mounting brackets arrived. Since the access points have not yet arrived, they decided to do a pilot installation using the equipment previously taken from Cisco Systems for the radio survey.
In the meantime, the engineers are doing their best to work out the solution for SMS authentication of subscribers. A separate subsystem should be responsible for this. There are several software manufacturers and telecom operators on the market from which it is possible to buy this subsystem, either as software or as a service. We settled on a product called WNAM from LLC Netams, which worked with us at the Otkrytie Arena stadium. This is how it should look in general terms.
WNAM communicates with Wi-Fi controllers using the RADIUS protocol. When the subscriber tries to connect to the SSID, the controller asks WNAM if there is a client with the MAC address from which the connection request came in the registered user database. If there is - the client gets access to the Internet and optionally redirects to the starting website (in our case, this is the FIFA portal welcome2018.com). If it does not exist, the controller redirects the user to the authentication portal, and so far, except for this portal, it does not allow it. The portal is a web page that spins on the WNAM authorization server. The design of the authentication portal for each project is unique and is developed separately (in our particular case, it was sent by the FIFA Organizing Committee). During FM, the authentication start page should look like this:
As soon as the user enters his phone number, WNAM generates an authorization code and sends it as SMS via the kannel utility via the SMPP protocol to the SMS operator, and that via the appropriate cellular operator to the final subscriber.
In our case, in order to communicate with the operator, we need an IPSEC tunnel and, of course, an agreement under which this SMS operator will work with us.
For sending each SMS-message operator will be charged. The one who signs the contract, in this case the owner of the Wi-Fi infrastructure, must pay. (Actually, you can tell the subscriber a one-time code on the web portal, and then ask him to send this code to the number of the SMS operator. In this case, the subscriber will pay for sending the SMS, but we did not go that way).
Upon successful delivery of the SMS message, the cellular operator generates in the opposite direction a delivery report with confirmation of the fact of receiving the SMS. This message is transmitted by the SMS operator to WNAM, so we know whether the subscriber received a one-time password or not.
The user is prompted to enter a password on the portal. If the password is correct, WNAM instructs the controller to authorize the subscriber and give him access to the Internet.
February 1, 2018
Wi-Fi controllers, access switches and firewalls arrived at the site. The equipment was assembled and started up and commissioning. Colleagues from Netams started setting up WNAM. In the meantime, the customer was called from the relevant departments and was reminded of the high responsibility that rests with all of us in working out potential information security incidents at the World Cup matches.
To save time, we actively negotiate with the customer the pilot assembly units informally. We explain that the antennas and access points have already been purchased and are going to the site, so the maximum that we can change under extremely tight deadlines is to move points a little or slightly fix brackets. Otherwise, the project can be minimized. The customer goes forward and makes minimal adjustments to the pilot solutions. Thus, we slowly agree on the order of design batches of typical mounting brackets.
February 12, 2018
The organizing committee sent updated requirements for the work of the Wi-Fi network for fans. The main changes concern the statistical reports that the Organizing Committee will require from us at every World Championship match. Some requirements are very specific. In addition to the obvious things such as communication channel load schedules and the number of subscribers, it is required to calculate the ratio of users in the 2.4 and 5 GHz bands, as well as data on how subscribers go through various authentication stages, and even broken down by country from which these subscribers came . The FIFA Organizing Committee wanted to see statistics not only in the form of interactive graphs, but also in the form of tables with the ability to automatically export data to excel and pdf, and the option of setting a time period for which data is needed should be available before export. Such requirements suggested a separate “umbrella” system for collecting and analyzing statistics, which will summarize data from various service components, since information on the utilization of communication service provider channels, for example, must be taken from the firewall, authentication statistics — from WNAM logs, and data on the number of subscribers in the 2.4 and 5 GHz bands - only on controllers and nowhere else. So in our project appeared heavy artillery in the form of a system for collecting and analyzing statistics Splunk Enterprise. 4 and 5 GHz - only on controllers and nowhere else. So in our project appeared heavy artillery in the form of a system for collecting and analyzing statistics Splunk Enterprise. 4 and 5 GHz - only on controllers and nowhere else. So in our project appeared heavy artillery in the form of a system for collecting and analyzing statistics Splunk Enterprise.
February 14, 2018
But what to do with complaints about the poor performance of Wi-Fi? How to evaluate the work of the service, if you control not all of its components, there are too many "floating" problems, and the descriptions of incidents received from subscribers only take time? Why not scatter around the stadium some devices that will behave on the Wi-Fi network in the same way as our users, and at the same time send us detailed logs about how things are in reality? These should be devices that could connect to the Wi-Fi network themselves, authenticate to WNAM, work on the Internet, disconnect and do it all over the script again and again, sending us detailed logs at each step. Of course, all this should still be inexpensive, as the project budget is almost exhausted. It will be necessary to discuss this topic with the engineers, it may be possible to come up with something.
February 15, 2018
We discussed with engineers the idea of proactive monitoring of Wi-Fi by simulating user activity. Remember about the Raspberry Pi. In theory, it can be endowed with the functionality of a mobile device, if equipped with a GSM modem with a SIM card, which will receive SMS with a one-time authentication code. It is possible to feed a product on PoE, through a PoE-extractor.
February 26, 2018
Engineers issued a project of a proactive monitoring device based on the Raspberry Pi 3 Model B. It is supposed to connect the D-Link DWA-171 / RU / A1A Wi-Fi adapter to it, since the internal Wi-Fi adapter does not support networks in the range 5GHz (sorry that the B + version, where there is an integrated adapter for both bands, will be released a month later), as well as a Huawei E3372h-153USB 3G / GSM modem with a SIM card. The power supply of the minicomputer can be carried out via the Upvel UP-102S intermediate PoE extractor. All this will be packed in a plastic case.
On the minicomputer running the Raspbian OS, the Firefox Internet browser and the Splunk forwarder component are installed, which should send logs to Splunk Enterprise. The microcomputer activity scenario is defined by a separate python script. Logs are sent via wired Ethernet, not via Wi-Fi, which is another advantage of the solution.
The operation of the verification script is based on interaction with the browser and performing actions similar to those performed by the user in the process of working with Wi-Fi (loading the authentication page, entering the number on which the SIM card is registered, entering the received code via SMS, several speed tests on the Internet). For this ideally suited library Selenium, created for the language Python.
February 28, 2018
Customer liked the idea with Raspberry. He agreed to include 50 minicomputers in the solution and purchase three cellular operators for them SIM cards. Now, if during the match someone calls and starts complaining about the poor performance of Wi-Fi, we will have 50 reference devices, by which you can see the overall picture of the service at the stadium.
March 5, 2018
The first batch of mounting brackets arrived. All access points and antennas are already at the facility, the SCS cables are stretched, and there are not enough brackets. We mount what is. We agreed with the customer that by the Russia-Brazil test match, access points will be mounted on typical sectors in order to be able to verify the correctness of the decisions made for the placement. It is a pity that before the opening ceremony, we definitely will not have time to test the Raspberry.
March 15, 2018
We were invited to the next meeting on information security. At the meeting, we asked what IS risks during the Championship would be the most significant. It turned out that the most unpleasant thing that could happen was the broadcasting of extremist content to the video board of the stadium. We asked what would happen if someone thought of it during the match to display pornography on the scoreboard. We were told that this, of course, is also unpleasant, but there are worse things.
March 20, 2018
Today we have completed setting up traffic mirroring on the SORM-2, SORM-3 and SOPKA server interfaces, which are installed directly in the stadium data centers. I had to tinker a bit with redirecting the logs from WNAM and the firewall to SORM3, so that if there was any where to get the private IP address, MAC address and SIM number of anyone authorized in our network.
March 23, 2018. Test match "Russia - Brazil"
Today at the stadium there is a friendly match between the national teams of Russia and Brazil, so the installation of access points is suspended. For the test match, we managed to hang up access points only on the lower tier of the stands of the stadium (this is 50% of the total) plus Wi-Fi in the sub-bed space, which was already working. The main task for today is to make sure that our typical solution for the arrangement of access points will work in conditions of a large cluster of subscribers, and of course, the general maintenance of the IT infrastructure is also ours.
Our team is located on the ground floor of the eastern stands of the stadium in a room with white walls and no windows. There are only IT people, and we are about fifteen people. The people are sitting and staring intently at laptops. Someone initially lacks chairs. Still indoors it is stuffy and a lot of white light. All of this suggests thoughts of Orwell’s cameras in the Ministry of Love, but there’s no time to think about it.
Each of the engineers is responsible for the operation of a separate IT system: data storage, servers, virtualization, AD DS, Wi-Fi ... We have a separate table with the project manager and a large monitor that displays Splunk Enterprise monitoring systems. Splunk Enterprise collects logs from all IT systems in the stadium, for which we are responsible. It also shows the statistics of high-density Wi-Fi.
The statistics is very detailed, there is even a percentage of user distribution over the 2.4 and 5 GHz bands (they thought for a long time how to implement, and finally decided to periodically use Splunk via the API to access Wi-Fi controllers from Cisco and HPE through intermediaries in the face of the HP IMC monitoring system to which HP point controllers are connected, and Cisco Prime to which Cisco controllers are connected).
Since there are no windows in the room, we have no idea what is happening on the field. Therefore, in the corner of the room there is a TV that broadcasts the live broadcast of the First Channel. On this TV, we with the project manager will watch all the matches of the 2018 World Cup at the Luzhniki Stadium.
From the room where we sit, there is a short passage to one of the entrance turnstile groups. That is, you can go out and see how people pass through the turnstiles to the stadium. From this, in general, the everyday process is impossible to break away, because the turnstiles are synchronized with the ticket server on the network, for which you answer with your head. It is terrible to imagine what will begin if the turnstiles stand.
People are pulling up to the stadium. We see how they pass through the turnstiles, and we observe in Splunk dashboards how the number of connected users is growing. It grows evenly until the starting whistle, then falls, in the break between the first and second half, another peak, then a recession again, and finally a small final surge after the end of the match. No incidents have been recorded. The decision on the arrangement of access points in the stands of the stadium was recognized as workable.
April 16, 2018
As you know, in order to identify a person by phone number, you need to make a request to the telecom operator to obtain passport data of the person to whom the number is registered. If you have registered a SIM-card in Iran, Mexico, Morocco, it’s not a fact that you will be able to identify you promptly through a request to the cellular operator.
Therefore, the Organizing Committee sent the updated requirements for SMS authentication. If at the World Championships someone tries to register on a Wi-Fi network using a foreign number, they will be asked to enter a FAN-ID number or fan passport. The implementation of this requirement requires separate dances with a tambourine at the WNAM level in terms of integrating the SMS authentication system with the FAN-ID databases. Especially since the FAN ID portal with which to integrate is still under active development.
May 15, 2018
Added an authentication portal page for foreign SIM-cards to WNAM. Since the FAN ID database is not yet operational, the FAN ID authentication does not work yet.
Finally completed the installation and commissioning of all access points. The system is ready for operation. Unfortunately, we have not managed to mount the Raspberry yet, but there’s nothing really terrible about it. We will carry out these works until June 14.
June 12, 2018
Today we were offered to check how the Wi-Fi will work if the stadium launches electronic warfare against unmanned aerial vehicles. During the test, Wi-Fi revealed that the signal level has become much worse. We were left to work with electronic warfare equipment, and this resolved the issue. Thanks for the constructive attitude.
June 14th. Match "Russia-Saudi Arabia"
I am standing in the corner of the Luzhniki conference hall, and in the conference hall, preparations are underway for the meeting before the opening ceremony of the World Championship. Next to me is a box of beer, on which lies a piece of paper that says “Rider R. Williams. Do not take". The opening ceremony will begin in a few hours.
We have just finished the last thing we had to do at the stadium before the World Cup - we mounted and connected a TV panel on the presidential VIP-platform, which will display match statistics and repetitions of interesting moments. They could not resist and sat in the chairs in which Vladimir Putin, Crown Prince of Saudi Arabia Mohammed bin Salman and FIFA President Gianni Infantino would soon be sitting.
Each of us has accreditation to the stadium. To get it, each filled out a questionnaire and fingerprinted. Our accreditation gives the right to move freely around the stadium, but in the period three hours before the start of the match, during the match itself, as well as during the rehearsals of the events and training of the teams, we are not allowed to enter the field and stands. Therefore, three hours before the match, we go down to our headquarters and turn on the TV. Fans are slowly catching up. The number of subscribers who connect to Wi-Fi is gradually increasing.
The stadium's IT infrastructure completed the match without incident. However, the low percentage of users who logged into the SMS authentication portal and successfully passed it confuse - 30% with an expected 50%. We assumed that everything was due to the five goals scored by the Russian national team at the gates of the national team of Saudi Arabia, but the Organizing Committee does not agree with us. Let's see what will happen in the next matches.
June 15, 2018
We analyzed the WNAM logs and found out that during yesterday's match a lot of people for some reason, when registering on the WiFi HD portal, incorrectly entered their FAN ID for some reason. If you look at the passport of the fan, then there are three numbers, similar to those that require authentication portal page. Assumed that the fans did not know what exactly the number you want to enter. As a result, together with the FIFA Organizing Committee, we came up with a couple of ideas that should improve the quality of the Wi-Fi service at the stadium:
- add a fan passport image to the portal with an indication of what specific number you want to enter,
- add instructions for those who fail to connect,
- ask the stewards to contact us for technical support if someone from the fans turns to them for support.
June 17, 2018. Match "Germany - Mexico"
In the midst of the match from the stewards comes a complaint about the problems with access to Wi-Fi in the VIP-zone. We are escorted to the sky box. In the sky box, a table is set and a crowd of Mexicans hang out. In the corner - the bar. It is downright bursting with a variety of alcohol. Near the bar stand is a hefty Mexican. He has beer and a phone in his hands. It's hard for him to stand. He was the one who complained about problems with Wi-Fi.
We communicate in English. We explain that in this field the Mexican must enter his mobile number:
- Tell us your phone number. Do you remember your phone number?
Mexican staggers. At first, he doesn’t remember his phone number at all, but on the third attempt he still remembers it. We continue to explain:
- Now you must enter your FAN ID number. Show your fan passport, please.
“Listen,” he says , “I just want to post a selfie on Instagram.” In my opinion, for self it is all too difficult. So I’ll just watch football and get by without selfie and without you. You are very hospitable, goodbye.
Internet access for a Mexican, we still organized. And at the end of the match I was sent a video of how the fans poured beer on the steward. It is not easy to remember your phone number when the magic of Mundial is raging around you.
June 18, 2018
Every time after the match, we hold an internal meeting at which we discuss the outcome of the event, whether it is possible to improve the quality of the infrastructure and how to do it. This time the meeting was devoted to the decision on the Raspberry Pi, which during the match Russia-Mexico worked in full force, but surprised us with strange statistics. In the morning, when there was no one at the stadium, Raspberry informed us that Wi-Fi was working fine. However, as soon as the fans began to take their places, all minicomputers began to inform us about the complete lack of service. At the same time, Splunk and the WNAM reporting subsystem reported that the connections were going fine. It turns out that the data contradict each other.
During the step-by-step debugging of the Raspberry script, it turned out that it needs to be substantially refined, since in its current form it gives a huge amount of false errors. It was decided to release a new version of the script, which will contain 15 additional checks. Replaced Firefox's fairly heavy browser with PhantomJS, which ultimately made it possible to double the speed of checks. Another possibility of the system was to display on the pages of statistics in Splunk screenshots of the authentication page for visual assessment of possible problems.
To quickly fill the new versions of the script on 50 devices, we decided to install the Ansible orchestrator.
An analysis of the script also revealed completely unobvious things: for example, there was a moment when all the Raspberry started swearing at the wrong one-time passwords. It turned out that all microcomputers, in addition to the codes, received other SMS messages, for example, spam or messages from the Emergencies Ministry about the deterioration of the weather. In the scan cycle, such SMS messages were buffered before messages generated by WNAM, and, accordingly, were perceived by the script as a one-time password.
June 20, 2018. Match "Portugal-Morocco"
“What the hell is Morocco doing about this?” Exclaimed our engineer in charge of monitoring WNAM in a rush when he saw a growing queue of sending outgoing SMS messages without a delivery report. We in the logs had statistics on sending SMS-messages for all operators who have codes in open databases, and we could see which countries of the SMS arrive normally, and which ones - with a delay. The Kingdom of Morocco has a strong outsider position in our statistics. At the beginning of the second half, it was decided to redirect the newsletter to another SMS operator, and this somewhat improved the situation.
June 26, 2018. Match "Denmark-France"
On TV, the live broadcast of the match from the stadium was usual. We need to view the broadcast in order to feel the dynamics of the activity of Wi-Fi clients: during the game it falls, and during breaks it grows on the contrary.
Compared with the match “Portugal-Morocco”, the accompaniment turned out to be quieter, since Denmark and France - European countries and statistics by SMS were expectedly better for them than for Morocco. Therefore, we dedicated the match to debugging Rasberry scripts and became so enthusiastic about this process that only at the end of the match we understood: all this time a completely different match was broadcast on TV, not from the Luzhniki stadium, but from the Fisht stadium.
In the subway after the match, a peasant bore me with questions:
- Listen, and who just played at the Luzhniki Stadium?
I hardly remember the names of the teams:
- Denmark and France. Yes, France.
- What is the score?
I don’t remember the score at all, because I didn’t watch the match, and I reported it. When the man realized that I was going from the stadium and did not know the bill, he looked at me with unconcealed amazement.
July 11, 2018, the match "England - Croatia"
In the middle of the second half on the firewall, a suspicious network activity of one of the connected subscribers was discovered. The firewall blocked from its IP address a large number of connections on non-standard ports (which were closed by the firewall policy). Inquiries went to Russia, the CIS countries and Europe. Since this was an information security incident, security guards enthusiastically got down to business.
The WNAM logs have calculated the phone number to which the subscriber is registered. Called to the specified number. Introduced themselves to the Luzhniki service. They told some peasant that we were seeing suspicious network activity from the phone and therefore we would like to clarify if he is a hacker. The man at the other end of the line told us that he understood absolutely nothing. We said that in this case, if he does not object, we will block his access to the network. The man did not mind. There was some impression that he even supported this initiative. We blocked access and transferred the incident logs to the customer.
July 15, 2018, France-Croatia match, final
In the middle of the first half, a Luzhniki operation service representative came to our headquarters. He brought with him a spectator whom we called during the last match. It turned out that the man showed an active position, contacted the Luzhniki exploitation service and asked to arrange a meeting. He brought the phone with him and stated that he had not made any hacking attempts, so he was somewhat alarmed and would like to know what his complaints were.
He turned out to be an inexpensive Android smartphone from Samsung, a quick inspection of which revealed the torrent client installed on it, as well as many other software of dubious origin, which probably generated suspicious activity. We recommended a person to scan the phone with an antivirus, remove excess software and refrain from using the torrent client.
Conclusion
The participation of LANIT in the reconstruction of the IT infrastructure of the Big Sports Arena "Luzhniki" for the World Cup has ended. For four and a half years of work, we built about fifty low-voltage and IT systems and three data centers, hung up one and a half thousand cameras and 650 TV panels, set up a LAN at 12 thousand ports and made two high-density Wi-Fi networks for the Championship fans - 430 Cisco access points in the stadium bowl and 650 HP access points in the sub-tribune space.
What non-obvious features were revealed in the solution during the commissioning and trial operation of HD WiFi on the BSA Luzhniki:
- SMS authentication is not an out-of-the-box solution. Deploying an effective system requires highly qualified customization and active monitoring.Это – разработка портала аутентификации, настройка взаимодействия с SMS-оператором, и на закуску самое сложное – настройка интеграции с контроллерами Wi-Fi. Хотя RADIUS – это стандартный протокол, у него есть свои особенности взаимодействия с каждой моделью контроллера. Некоторые модели контроллера вообще не получится интегрировать. Нам повезло: с контроллерами Cisco 8510 и HPE870 WNAM заработал, причем дружить с HPE870 он отказывался практически до последнего. За счет интеграции с WNAM обоих контроллеров хоть как-то удалось решить задачу роуминга между сетями Wi-Fi на Cisco и HPE. Если человек переходил от зоны действия одной сети к зоне действия другой, сессии, конечно, обрывались, но за счет наличия общей базы данных абонентов переключение проходило более-менее незаметно.
- SMS delivery does not always work as desired. Especially when it comes to exotic countries.Есть куча причин, по которым SMS может прийти слишком поздно (мы знаем пару венгров, которые ждали SMS с кодом по 5 минут), либо не прийти вообще. Одна их них — спам-фильтры на стороне SMS операторов. Например, одноразовый код доступа в SMS практически гарантированно не придет абоненту с иностранной SIM-картой, если идентификатор отправителя будет представлять собой не номер телефона, а имя (например, Luzhniki). Последовательность из нескольких одинаковых сообщений одному и тому же номеру тоже может быть воспринята оператором как спам и заблокирована. Проблема усугубляется тем, что иногда сообщение передает не один, а цепочка неизвестных вам SMS-операторов, у каждого из которых – свои настройки спам-фильтров.
- A subscriber may refuse to authenticate for obvious reasons.Не всем нравится идея сообщать номер своего мобильного телефона в обмен на доступ в интернет. Нормальной считается ситуация, когда доступ в интернет получает 50% от тех, кто попадал на портал аутентификации. Тем не менее, для повышения качества сервиса всегда полезно выяснять причины, по которым абоненты пытались зарегистрироваться в сети, но в Интернет не попали. Возможно, с этим удастся что-то сделать.
- HD Wi-Fi fans need at the stadium much less than we would like.Когда мы проектировали HD Wi-Fi, мы рассчитывали, что им одновременно воспользуется 20% болельщиков, хотя на всех матчах показатели оказались ниже этого значения. Среди методов, которыми мы пользовались для увеличения числа пользователей, сработали: голосовые и текстовые объявления о том, что на объекте работает бесплатный Wi-Fi, а также добавление на портал аутентификации детальной инструкции по подключению.
- You will never be the only one distributing Wi-Fi on site.Во время последнего матча мы запустили сканирование сети на предмет Rogue AP и только на трибунах стадиона насчитали более 3000 «чужих» устройств, работающих в режиме точки доступа. Причем это были как сотовые телефоны, раздающие Wi-Fi, так и специальные решения зарубежных операторов, например, Skyroam и Roamingman. Возле входных турникетов висело объявление, запрещающее раздавать на стадионе Wi-Fi, но не похоже, что оно хоть как-то работало.
- Proactive monitoring of Wi-Fi HD, imitating user activity, can be effective, but it requires a lot of effort in implementing and maintainingНамного веселее получать данные от микрокомпьютеров Raspberry, чем сидеть и ждать очередной жалобы от недовольных пользователей. Разумеется, Raspberry капризничали, а написание и отладка скриптов заняли у нас много времени. Зато это позволило взглянуть на сервис Wi-Fi HD с другого ракурса, направило процесс мониторинга в более конструктивное русло и в итоге действительно помогло нам локализовать несколько «плавающих» ошибок, что позитивно сказалось на качестве сервиса. К тому же, всякий раз, когда кто-то обращался к нам с вопросами о работе Wi-Fi HD у его девушки, нам было что ему ответить.