July 18, 2014 at 20:09Lumia on a Midsummer Night: how the photo installation was created at the MSND festival
On June 21, the day of the summer solstice, in the Serednikovo estate near Moscow, the sixth costume ball Midsummer Night's Dream was held . Every year, guests in truly impressive fancy dress costumes gather at the event. However, today we want to tell a little about this.
Our cooperation with the organizers has become a good tradition. This year, we decided to surprise visitors with an interactive installation - “bullet time” with a photographic machine consisting of 60 Nokia Lumia 1520 smartphones. It allowed us to take impressive live pictures with the effect of a frozen time in 360 ° format, which the festival guests could later enjoy. We talked about the technical aspects of creating this photographic machine with our colleagues who were responsible for the direct implementation of the project.
We came to our colleagues from Hello Computer with a ready-made idea and ... tight deadlines. Fortunately, they readily took up the work:
“ Only three weeks were allotted for the implementation of the project, and after all the approvals and drafting of the ToRs, in general two. Therefore, we immediately tried to determine the most optimal solutions and platforms that could provide us with fast development speed.
First of all, we started by synchronizing the phones. It's clear that getting 60 devices connected wirelessly to take a snapshot at the same time is quite a challenge. Therefore, our choice fell on the client-server architecture . ”
As planned by the developers, the phones (clients) were supposed to contact the server, synchronize the time with it, calculating the corrections for the signal passage and get the time from the server when it was necessary to take a picture.
The team that worked on the project included 7 people: two Windows / Windows Phone developers (one worked as an application for the phone, the second on the tablet); server developer working with a central server and tools for gluing and sharing; project producer and producer assistant responsible for management; designer; technical director, on whose shoulders lay down the compilation of technical specifications, coordination of work, prototyping, designing of the network and equipment.
As expected, the collective thought process began with heated discussion, though the Nokia Lumia 1520 itself became the subject:
“ In our studio, only one person uses Windows Phone, so everyone else was interested in testing the devices and at the same time spending half a day on the“ holivar ”on the following topics:“ Do you need such large phones? ”,“ Is the Windows Phone platform good? ” etc. ".
However, soon the project began to take on its features from the technical side:
“ Since we decided on the architecture of the project from the very beginning, the main task for us was to remove as much logic as possible on the server. Setting any kind of settings in 60 phones is a very laborious task.
We started with a simple prototype server on NodeJS and a skeleton application for a phone on Windows Phone 8.1. Once a second, the server sent requests to smartphones to synchronize the current time and the time of the moment of shooting. We also concluded that the difference between the “internal” time of the device and the timing that it received from the server. This difference pleased us: it was only within 20-30 milliseconds . "
At the first stage of design, the developers had 5 smartphones, of which they collected the first prototype. After a quick test, as usual, unforeseen difficulties became clear :
“ The first thing that caught my eye was the difference in color and illumination of the images. And for the “gluing” we needed to get the same picture on all devices.
Here, Windows Phone pleased us with good and complete documentation (for example, about what settings can be transferred to the phone) and ready-made examples. It was important for us to set the same white balance, shutter speed and ISO on all devices. And thanks to the advanced shooting settings in Nokia Lumia 1520, which are usually found only in cameras, we were able to easily frame the frame on each of the devices. We were also pleasantly surprised and the manual focus adjustment mode turned out to be useful.
The phones also received all these settings from the server . ”
Next, the team had to establish interaction between devices and the server:
“Pretty quickly, we set up automatic sending of photos back to the server. At this point, we had a ready-made prototype application that could begin to be tested on more phones.
We organized all client-server interaction according to the principles of the REST API. In NodeJS (more precisely, in Express) this is done quite quickly and simply. We made the management interface in the form of a simple web page that opens from the local server. Data exchange was implemented using web sockets. Fortunately, we didn’t have to worry about cross-browser compatibility . ”
In parallel with writing an application for phones, the team was working with utilities for gluing photos into one video with their subsequent placement on the network. As was clear from the number of devices involved in the project, the utility should have received a set of 60 photos at the input.
“ We decided not to reinvent the wheel and used ffmpeg for gluing and resizing photos. He pleasantly surprised us by the fact that he could do everything we needed: stick together a photo in a video with a change in compression quality and fps ;, resize the photo, put a watermark on it. ”
An important task of the project was the ability to automatically post the resulting video on the network. For us, the social component of this project was a priority, therefore, the team was required to teach the photographic machine to automatically upload video to a public stream, send links to e-mails and enable guests of MSND to share video on social networks.
“ As a video hosting, it was decided to take Coub- for several reasons. First of all, our short looped music videos are the perfect format for Cobe. Secondly, this service has a convenient API for loading, with which we have already worked before. The documentation, unfortunately, is hidden, but the guys send it on request. Thirdly, here it was possible to easily and conveniently configure the feed and beautifully arrange it. Well, of course, each video could be shared on the network - this feature was provided by Cow himself, which saved development time.
From the previous project, we still have a suitable Python script. It worked quite simply: it scanned the desired folder and uploaded each new video file, after which it created a small text file with the video address and its status. We specifically refused to use databases in favor of script flexibility and portability to any other project. Also this script was able to send video to e-mail . "
We also asked to add the ability to download video to our device through the OneDrive service:
“OneDrive has a clear and convenient API. Therefore, we simply supplemented our script with a module for uploading video to the "cloud". First, he went through standard OAuth authorization, then uploaded files to our account and received a link for viewing and downloading. This link was saved in the same small file . "
An important aspect of the work of all utilities was support for offline operation. If the download of files for one reason or another failed, the scripts quietly tried to repeat it after a while.
As a device on which guests could view their 360 ° pictures, a Windows RT tablet was chosen - Lumia 2520. The developers wrote an application that also accessed the server and received from it a list of all the videos. In the application, users could enter mail and also receive a QR code with a link to OneDrive or Coub.
When the software was ready, it was time to move on to field trials. To begin with, all 60 smartphones and the server were combined into one network. In order for the team not to get confused during the final assembly and further work with content, the server was assigned 254 addresses, and smartphones from 1 to 60. All devices were strictly numbered, and each MAC address was assigned its own IP address in the router.
However, the simultaneous connection of 60 phones to one Wi-Fi network turned out to be not such an easy task:
“ We got all 60 smartphones 5 days before the launch, and here the funniest“ dancing with a tambourine ”began in this project.
We started tests on one of the top routers (hereinafter in the text manufacturers and models of routers are not mentioned in order to avoid any reputational damage, author's note). He calmly withstood the connection of all 60 Lumia, but when trying to get pictures from devices, the files started to get lost. We were ready for this and had ready "plates" from another well-known manufacturer in order to evenly distribute clients across three routers . "
However, in this case too, the team failed: the transfer of the first package of photographs took place without any problems, but the rest were lost during the transfer or arrived with a delay of several minutes.
"We started to panic - 4 days before launch, and with Wi-Fi everything was bad. They tried to breed everything in different disjoint channels. It became better, but not by much. Despite the fact that the "weight" of one photo is only 500 kilobytes, that is, the total size of 60 photos was 30 megabytes + service traffic . "
A long phase of testing has begun. Engineers tried to understand how the situation changes when a certain number of phones is disconnected.
“ Probably, the conclusions we came to seem logical for specialists, but we faced such problems for the first time. Therefore, their solution cost us a lot of time.
The key mistake for us was that we tried to transfer all the photos at once. The signals interfered with each other, so the files were transmitted very slowly. We were able to solve the problem by setting a delay of 2 seconds for each next device in the chain . ”
As a result, we came to a scheme in which every 20 devices were allocated to a separate router and its own separate channel. This allowed us to transfer 1 file at a time to each router taking into account a two-second delay without any problems. In such an architecture, sending a single image took from 0.5 to 2 seconds (sometimes there were delays of up to 10 seconds), but file loss was already excluded from the list of problems.
"Thus, collecting all the photos from the devices took about 45 seconds. Probably, this process could be done faster, but in the conditions of a tough deadline there was no time for further experiments. Yes, and such a speed, in principle, suited everyone: changing guests inside the "farm" (the internal name of the installation among developers, author's note) and preparing for the shooting took about the same time . "
The “farm” was based on the classic design for shooting video in bullet time format: a curved spiral-shaped frame on which shooting devices are mounted. During the discussion of the project, some decorative improvements were proposed. One of them was, for example, a decorative bonfire suitable for the theme of the event in the installation center, through which guests had to jump at the time of shooting.
Experimenting with the shape of the frame, the team settled on a four-meter radius and a vertical arrangement of phones. This was explained both by the peculiarities of shooting the smartphones themselves and by the budget: an increase in the radius to 6 or 8 meters would lead to a significant increase in the cost of the project.
"The main engineering task at this stage was the development of the mount, which would allow very fine-tuning the position of the phones in 2 rotation planes. As a result, we settled on a plastic mount with metal clips. The only thing left was to produce and assemble everything on time . ”
At the last stage, the team had to assemble disparate components into one single system. They only had two days to install the equipment. The first day was fully devoted to the installation of construction and Internet equipment, and the second to the installation and configuration of smartphones, technical equipment and their testing.
“ The most difficult and time-consuming stage is the installation of smartphones. Filling software on all 60 devices takes about an hour. Now imagine what any bugs found cost us (it’s good that the client didn’t agree to 120 phones :)).
At the testing stage, we actually encountered errors several times, as a result, we had to update the software on all devices again and again. Probably, it would be possible to make an update system by air, but there was no time for this.
Another story - the installation of 60 "Lumiy". Smartphones had to be mounted very precisely so that there were no jumps in the final videos. To do this, we made a special setup mode when smartphones displayed the image from the camera on the screen with a mark in the center. In the center of the "farm" we installed a stand with a bright target, which smartphones were pointing at. At first glance, it seems that it’s simple, but in reality it took us about 4 hours to initial setup and another 3 hours to adjust . ”
All night and the next day, until the start of the event, the team was engaged in testing software, debugging modules and fixing software errors.
Realizing a project of this level in a two-week period was a serious challenge for the development team.
“ This project was made for a specific event and had a clear“ deadline ”, so a delay of even a day would be a complete failure for us all. We needed to clearly plan our resources.
We tried first to simplify the task as much as possible. Make a raw minimum, and then add “features” and additional functionality to it. And so in all aspects - from programming to design design.
In order to precisely meet the deadline, all processes had to be launched in parallel. We tried to make the most autonomous components so that the developers were not dependent on each other . "
“ In fact, such projects charge a lot. We worked very closely throughout all two weeks. It was felt that everyone was rooting for the result. Including Nokia: the client went to meet us and promptly made all decisions.
In general, the project was a success, and we implemented everything on time. Of course, there are many more aspects that can be improved.
For example, we were not able to achieve super accurate synchronization: some people in the video jumped. In this case, you need to come up with some more reliable solution. We also have to work on the network and improve the speed of taking photos, making it a separate process in the background and, possibly, transferring the system to the 5 GHz standard.
As for the Windows Phone and Windows OS installed in the Nokia Lumia 1520 on the tablet, we were generally satisfied with them. Not being closely acquainted with the development for these platforms, we were skeptical about them. As it turned out, not only are they convenient and enjoyable to work with, Microsoft and Nokia are building a good infrastructure around them with tools and complete documentation . ”
Our cooperation with the organizers has become a good tradition. This year, we decided to surprise visitors with an interactive installation - “bullet time” with a photographic machine consisting of 60 Nokia Lumia 1520 smartphones. It allowed us to take impressive live pictures with the effect of a frozen time in 360 ° format, which the festival guests could later enjoy. We talked about the technical aspects of creating this photographic machine with our colleagues who were responsible for the direct implementation of the project.
Process
We came to our colleagues from Hello Computer with a ready-made idea and ... tight deadlines. Fortunately, they readily took up the work:
“ Only three weeks were allotted for the implementation of the project, and after all the approvals and drafting of the ToRs, in general two. Therefore, we immediately tried to determine the most optimal solutions and platforms that could provide us with fast development speed.
First of all, we started by synchronizing the phones. It's clear that getting 60 devices connected wirelessly to take a snapshot at the same time is quite a challenge. Therefore, our choice fell on the client-server architecture . ”
As planned by the developers, the phones (clients) were supposed to contact the server, synchronize the time with it, calculating the corrections for the signal passage and get the time from the server when it was necessary to take a picture.
Command
The team that worked on the project included 7 people: two Windows / Windows Phone developers (one worked as an application for the phone, the second on the tablet); server developer working with a central server and tools for gluing and sharing; project producer and producer assistant responsible for management; designer; technical director, on whose shoulders lay down the compilation of technical specifications, coordination of work, prototyping, designing of the network and equipment.
Architecture
As expected, the collective thought process began with heated discussion, though the Nokia Lumia 1520 itself became the subject:
“ In our studio, only one person uses Windows Phone, so everyone else was interested in testing the devices and at the same time spending half a day on the“ holivar ”on the following topics:“ Do you need such large phones? ”,“ Is the Windows Phone platform good? ” etc. ".
However, soon the project began to take on its features from the technical side:
“ Since we decided on the architecture of the project from the very beginning, the main task for us was to remove as much logic as possible on the server. Setting any kind of settings in 60 phones is a very laborious task.
We started with a simple prototype server on NodeJS and a skeleton application for a phone on Windows Phone 8.1. Once a second, the server sent requests to smartphones to synchronize the current time and the time of the moment of shooting. We also concluded that the difference between the “internal” time of the device and the timing that it received from the server. This difference pleased us: it was only within 20-30 milliseconds . "
At the first stage of design, the developers had 5 smartphones, of which they collected the first prototype. After a quick test, as usual, unforeseen difficulties became clear :
“ The first thing that caught my eye was the difference in color and illumination of the images. And for the “gluing” we needed to get the same picture on all devices.
Here, Windows Phone pleased us with good and complete documentation (for example, about what settings can be transferred to the phone) and ready-made examples. It was important for us to set the same white balance, shutter speed and ISO on all devices. And thanks to the advanced shooting settings in Nokia Lumia 1520, which are usually found only in cameras, we were able to easily frame the frame on each of the devices. We were also pleasantly surprised and the manual focus adjustment mode turned out to be useful.
The phones also received all these settings from the server . ”
Next, the team had to establish interaction between devices and the server:
“Pretty quickly, we set up automatic sending of photos back to the server. At this point, we had a ready-made prototype application that could begin to be tested on more phones.
We organized all client-server interaction according to the principles of the REST API. In NodeJS (more precisely, in Express) this is done quite quickly and simply. We made the management interface in the form of a simple web page that opens from the local server. Data exchange was implemented using web sockets. Fortunately, we didn’t have to worry about cross-browser compatibility . ”
Bonding and networking
In parallel with writing an application for phones, the team was working with utilities for gluing photos into one video with their subsequent placement on the network. As was clear from the number of devices involved in the project, the utility should have received a set of 60 photos at the input.
“ We decided not to reinvent the wheel and used ffmpeg for gluing and resizing photos. He pleasantly surprised us by the fact that he could do everything we needed: stick together a photo in a video with a change in compression quality and fps ;, resize the photo, put a watermark on it. ”
An important task of the project was the ability to automatically post the resulting video on the network. For us, the social component of this project was a priority, therefore, the team was required to teach the photographic machine to automatically upload video to a public stream, send links to e-mails and enable guests of MSND to share video on social networks.
“ As a video hosting, it was decided to take Coub- for several reasons. First of all, our short looped music videos are the perfect format for Cobe. Secondly, this service has a convenient API for loading, with which we have already worked before. The documentation, unfortunately, is hidden, but the guys send it on request. Thirdly, here it was possible to easily and conveniently configure the feed and beautifully arrange it. Well, of course, each video could be shared on the network - this feature was provided by Cow himself, which saved development time.
From the previous project, we still have a suitable Python script. It worked quite simply: it scanned the desired folder and uploaded each new video file, after which it created a small text file with the video address and its status. We specifically refused to use databases in favor of script flexibility and portability to any other project. Also this script was able to send video to e-mail . "
We also asked to add the ability to download video to our device through the OneDrive service:
“OneDrive has a clear and convenient API. Therefore, we simply supplemented our script with a module for uploading video to the "cloud". First, he went through standard OAuth authorization, then uploaded files to our account and received a link for viewing and downloading. This link was saved in the same small file . "
An important aspect of the work of all utilities was support for offline operation. If the download of files for one reason or another failed, the scripts quietly tried to repeat it after a while.
The tablet
As a device on which guests could view their 360 ° pictures, a Windows RT tablet was chosen - Lumia 2520. The developers wrote an application that also accessed the server and received from it a list of all the videos. In the application, users could enter mail and also receive a QR code with a link to OneDrive or Coub.
Network architecture
When the software was ready, it was time to move on to field trials. To begin with, all 60 smartphones and the server were combined into one network. In order for the team not to get confused during the final assembly and further work with content, the server was assigned 254 addresses, and smartphones from 1 to 60. All devices were strictly numbered, and each MAC address was assigned its own IP address in the router.
However, the simultaneous connection of 60 phones to one Wi-Fi network turned out to be not such an easy task:
“ We got all 60 smartphones 5 days before the launch, and here the funniest“ dancing with a tambourine ”began in this project.
We started tests on one of the top routers (hereinafter in the text manufacturers and models of routers are not mentioned in order to avoid any reputational damage, author's note). He calmly withstood the connection of all 60 Lumia, but when trying to get pictures from devices, the files started to get lost. We were ready for this and had ready "plates" from another well-known manufacturer in order to evenly distribute clients across three routers . "
However, in this case too, the team failed: the transfer of the first package of photographs took place without any problems, but the rest were lost during the transfer or arrived with a delay of several minutes.
"We started to panic - 4 days before launch, and with Wi-Fi everything was bad. They tried to breed everything in different disjoint channels. It became better, but not by much. Despite the fact that the "weight" of one photo is only 500 kilobytes, that is, the total size of 60 photos was 30 megabytes + service traffic . "
A long phase of testing has begun. Engineers tried to understand how the situation changes when a certain number of phones is disconnected.
“ Probably, the conclusions we came to seem logical for specialists, but we faced such problems for the first time. Therefore, their solution cost us a lot of time.
The key mistake for us was that we tried to transfer all the photos at once. The signals interfered with each other, so the files were transmitted very slowly. We were able to solve the problem by setting a delay of 2 seconds for each next device in the chain . ”
As a result, we came to a scheme in which every 20 devices were allocated to a separate router and its own separate channel. This allowed us to transfer 1 file at a time to each router taking into account a two-second delay without any problems. In such an architecture, sending a single image took from 0.5 to 2 seconds (sometimes there were delays of up to 10 seconds), but file loss was already excluded from the list of problems.
"Thus, collecting all the photos from the devices took about 45 seconds. Probably, this process could be done faster, but in the conditions of a tough deadline there was no time for further experiments. Yes, and such a speed, in principle, suited everyone: changing guests inside the "farm" (the internal name of the installation among developers, author's note) and preparing for the shooting took about the same time . "
"Farm"
The “farm” was based on the classic design for shooting video in bullet time format: a curved spiral-shaped frame on which shooting devices are mounted. During the discussion of the project, some decorative improvements were proposed. One of them was, for example, a decorative bonfire suitable for the theme of the event in the installation center, through which guests had to jump at the time of shooting.
Experimenting with the shape of the frame, the team settled on a four-meter radius and a vertical arrangement of phones. This was explained both by the peculiarities of shooting the smartphones themselves and by the budget: an increase in the radius to 6 or 8 meters would lead to a significant increase in the cost of the project.
"The main engineering task at this stage was the development of the mount, which would allow very fine-tuning the position of the phones in 2 rotation planes. As a result, we settled on a plastic mount with metal clips. The only thing left was to produce and assemble everything on time . ”
X day: installation
At the last stage, the team had to assemble disparate components into one single system. They only had two days to install the equipment. The first day was fully devoted to the installation of construction and Internet equipment, and the second to the installation and configuration of smartphones, technical equipment and their testing.
“ The most difficult and time-consuming stage is the installation of smartphones. Filling software on all 60 devices takes about an hour. Now imagine what any bugs found cost us (it’s good that the client didn’t agree to 120 phones :)).
At the testing stage, we actually encountered errors several times, as a result, we had to update the software on all devices again and again. Probably, it would be possible to make an update system by air, but there was no time for this.
Another story - the installation of 60 "Lumiy". Smartphones had to be mounted very precisely so that there were no jumps in the final videos. To do this, we made a special setup mode when smartphones displayed the image from the camera on the screen with a mark in the center. In the center of the "farm" we installed a stand with a bright target, which smartphones were pointing at. At first glance, it seems that it’s simple, but in reality it took us about 4 hours to initial setup and another 3 hours to adjust . ”
All night and the next day, until the start of the event, the team was engaged in testing software, debugging modules and fixing software errors.
Planning
Realizing a project of this level in a two-week period was a serious challenge for the development team.
“ This project was made for a specific event and had a clear“ deadline ”, so a delay of even a day would be a complete failure for us all. We needed to clearly plan our resources.
We tried first to simplify the task as much as possible. Make a raw minimum, and then add “features” and additional functionality to it. And so in all aspects - from programming to design design.
In order to precisely meet the deadline, all processes had to be launched in parallel. We tried to make the most autonomous components so that the developers were not dependent on each other . "
Impressions
“ In fact, such projects charge a lot. We worked very closely throughout all two weeks. It was felt that everyone was rooting for the result. Including Nokia: the client went to meet us and promptly made all decisions.
In general, the project was a success, and we implemented everything on time. Of course, there are many more aspects that can be improved.
For example, we were not able to achieve super accurate synchronization: some people in the video jumped. In this case, you need to come up with some more reliable solution. We also have to work on the network and improve the speed of taking photos, making it a separate process in the background and, possibly, transferring the system to the 5 GHz standard.
As for the Windows Phone and Windows OS installed in the Nokia Lumia 1520 on the tablet, we were generally satisfied with them. Not being closely acquainted with the development for these platforms, we were skeptical about them. As it turned out, not only are they convenient and enjoyable to work with, Microsoft and Nokia are building a good infrastructure around them with tools and complete documentation . ”