Opportunities for using 360 ° video technology to present cultural events
As part of the work at the Center for Design and Multimedia at ITMO University, we wrote an interesting article on the application of virtual reality technology to preserve cultural heritage. They described many of their own "cases" - museums, concerts, cultural events, and also told about a unique project of an interactive tour of the Korela fortress (Leningrad region), where they combined for the first time 3D-reconstruction and interactive "video 360".
In the English version, the materials will be presented in May at the international conference "ArtsIT, Interactivity & Game Creation" (Esbjerg, Denmark). Here is the Russian, slightly adapted original.
Virtual reality and its application
Currently, virtual reality technologies and their applications are being intensively developed. Great opportunities are opened by the use of such technologies in culture and art , as they provide the effect of the presence of the viewer in the center of cultural events. One of the most promising and currently actively developed approaches to creating content for virtual reality systems is the 360 ° video technology, which allows you to create panoramic videos with varying degrees of interactivity, where the viewer controls the viewing angle of the video at will. This video can be viewed as in a virtual reality helmet (for example, in the Oculus Rift , Samsung Gear VR , HTC Viveand others), and with the help of a special application on the smartphone, while the picture changes in accordance with the turns of the user's head. You can view 360 ° video on the display of a personal computer. In the latter case, the user controls the view using the mouse or keyboard.
Watching video in 360 ° format in Samsung Gear VR headset
Many large companies (Facebook, Nokia, Samsung, Google, etc.) are currently developing cameras for shooting 360 ° video, virtual reality headsets for various smartphones and desktops, as well as various sound recording devices that provide the creation of surround (binaural) sound and allow you to realize a whole complex of 360 ° multimedia technologies.
The use of 360 ° video technology in education is especially important. Interactive video 360 ° can provide interaction with the end user, so this method can be widely used to train specialists in various fields where the development of practical experience requires a presence at the center of the event. The 360 ° video technology allows you to analyze the teamwork of staff during the performance of various tasks, which leads to an increase in the quality of the educational process.
A significant role is played by a 360 ° panoramic video to preserve and ensure access to cultural heritage. This technology provides a unique opportunity to see with the immersion effect not only closed for visiting archives of museums, but also reconstructed historical monuments that were destroyed by time or circumstances (three-dimensional virtual reconstruction). In addition, virtual reality is a unique opportunity for people with disabilities to travel and walk around the world.
Speaking about entertainment in the aspect of technology video 360 ° - viewers can visit the largest cultural festivals, find themselves on stage with their favorite musicians or watch a theatrical performance from the royal box.
The ITMO University Design and Multimedia Center is implementing a project for the development and application of 360 ° video technologies in culture and art Video360Production.com . Within the framework of this project, it was possible to achieve certain results necessary for obtaining high-quality content, starting with the development of original video recording devices 360 ° and sound and ending with the development of software for solving a number of tasks in this area.
Multimedia technologies 360 °
Special cameras are currently being developed with the help of which one could shoot video in 360 ° video format. However, with the help of special tools and software, you can use existing cameras to shoot 360 ° video.
A striking example of this approach is the wholea number of projects offering special rigs for GoPro HERO 4 cameras. The number of cameras in the riga is from 6 to 14 (stereo versions), depending on the tasks. After shooting, video streams are stitched into the general video sphere using separately supplied programs.
In addition, several large IT companies and electronics manufacturers announced their own fully automated solutions for shooting content for virtual reality - Google Jump + GoPro Odyssey , Samsung Beyond , Nokia Ozo .
GoPro Odyssey, Nokia OZO and Samsung Project Beyond
We described our personal impressions of the Nokia OZO camera here .
The 360 ° video is a new and progressively developing direction that still does not have standards of direction, which provides a wide scope for creativity and the development of new ideas and approaches. The effect of full immersion is achieved by a camera with multiple lenses, which allows you to shoot everything that happens around. However, this entails many difficulties, because many laws of direction do not apply when shooting in 360 ° video format. The main cardinal difference in viewing 360 ° video in virtual reality headsets is the fact that the viewer has the opportunity to watch content around himself, which means that the director will have to provide the viewer with the ability to dynamically select the viewing angle and not focus on a specific part of the screen.
The solution to this problem is the detailed reasoning of the scenario from the point of view of a paradigm shift - the departure to 3D video space. Each shooting requires both standard preparation (setting up the light, the most advantageous points for the camera, scenario and positioning of the scene), and non-standard approaches (special and often unique for each camera attachment, direction taking into account the features of 360 ° video recording, correct camera orientation, surround sound recording). When recording concerts, to achieve greater realism, the sound from the camera or from the stereo microphone is often combined with the sound from the director's console.
Representation of cultural events using 360 ° multimedia technology
Video360production.com collection of video footage for virtual reality headsets includes more than fifty different videos: concerts and music festivals, tours of historical places, tours in museums, theatrical performances, various cultural events and technological processes.
Group "Bravo", "Love does not burn" in the format of panoramic video
Grouping "Leningrad" in the format of panoramic video
The technology of laying asphalt "Ecohiter", panoramic promotional video
When recording concerts, it is necessary to transmit in high quality, both image and sound. Moreover, it is necessary to think about the location of the camera, since when recording to a 360 ° camera, such concepts as “front” and “background” become irrelevant. In addition, sometimes the scene was simultaneously up to twenty people, which complicated the installation of the camera. The footage allowed the musicians to look at themselves from the outside, to assess the location of the musicians during the concert, as well as to see the public reaction to the performance. The spectators, in turn, with the help of this survey were able to examine in detail the play of each musician and immerse themselves in the atmosphere created by the orchestra or group.
A concert of the Picnic group in the format of a panoramic video.
During the shooting of the famous Russian group Picnic, the following difficulties arose: the concert was held in a dark room of the concert hall under the bright light of spotlights. These conditions are difficult for ordinary cameras, because even at maximum resolution, the transmission of realistic color depth is quite a challenge.
Video recordings of concerts in the 360 ° format have a huge advantage over standard video recordings, as the viewer feels the effect of being present in the stage space and has complete autonomy to decide for himself where to look at the moment — whether to watch the guitarist or look at the soloist, or maybe see concert "eyes" of musicians, turning his back to the stage, and watch the lively reaction of the public.
To achieve the effect of complete immersion, not only a high-quality picture is necessary, but also a “live” sound. For musicians and frequent visitors to concerts, it is not a secret that the “technical” sound on the stage is far from what is heard in the hall. The sound engineers of the project combined the sound from the stereo microphone with the studio recording. As a result, the sound remained high quality, but it retained its “vividness”: while watching the recording, you can hear how the audience reacts and sings.
In addition to musical groups, the performers of the project video360production.com cooperates with the largest St. Petersburg theaters, where a number of performances and musical performances were shot. A unique opportunity to provide shooting historical sites in cities and excursions in museums. In the Museum of Anthropology and Ethnography. Peter the Great (Kunstkamera) in a 360 ° video format, before the reconstruction began, a large academic ( Gottorp ) globe was filmed - a diplomatic gift to Tsar Peter I from the Duke of Golstinsky Karl Friedrich during the Northern War.
Shooting a large academic (Gottorp) globe in 360 ° video format
The implementation of a joint project of the ITMO University and the State Hermitage Museum “Points of Presence” has begun, in which it is planned to present events in various areas of the museum using panoramic video filming technologies.
The shooting of the Peacock Clock video in the State Hermitage in the format of 360 ° video.
A virtual walk along the rivers and canals of St. Petersburg is also of heightened interest among YouTube channel visitors created by the authors of the project.
Particular attention was paid to filming a 360 ° video of various events in the Solovetsky Monastery. The Solovetsky Monastery played an exceptional role in the history and culture of Russia. Originating in the XV century. as a place of desertification of spiritual followers of the "father of Russian monasticism" Sergius of Radonezh, in a short time he became the spiritual, cultural, political and economic capital of the North of Russia. By the works of the Solovetsky monks, who gave the Russian Church a whole galaxy of holy ascetics, the Christian faith spread to the endless expanses of the Russian Polar region.
The activity of the Solovetsky Monastery decisively contributed to the influx of the Russian population to the White Sea region. The marine monastery had a decisive impact on the development of the economy and the marine part of Pomerania. This was a prerequisite for the development of the Arctic Ocean by Russian sailors and the transformation of Russia into a great Arctic state.
The geographical remoteness of the Solovetsky Monastery makes it difficult for many to take a personal part in the events taking place there. Shooting such events in the technology of video 360 ° and providing access to them via the Internet allows, when using virtual reality gadgets, to feel like a participant in such events.
As a result of the work done, more than 10 scenes were filmed on the Solovetsky Archipelago: Excursion to the Ascension Church on Sekirnaya Hill, Service in the temple of the Solovetsky Monastery, Christmas Liturgy, All-night Vigil, Children's Christmas Festival in the Solovetsky Monastery, Monastic caroling, various kinds of the monastery and the Big Solovetsky Island.
Service in the temple of the Solovki monastery in 360 ° video format
Excursion to the Church of the Ascension on Sekirna Gora (Solovetsky Islands) in 360 ° video format
Interactivity in 360 ° video format
Despite the fact that viewing 360 ° 3D video in various virtual reality headsets provides quality virtual immersion in video content; the next step is the possibility of introducing various interactive elements into the video format of 360 °.
Such elements can be:
• Active tags inside the virtual space for movement along various trajectories previously captured in 360 ° video technology;
• Introduction to 360 ° video of various additional content (images, video, hyperlinks, etc.) - the function “picture in picture”;
• Transition from video in 360 ° video format to simulated 3D space of reconstructed reality.
On the basis of the Unity3D system in the Center for Design and Multimedia of ITMO University an interactive tour in the format of 360 ° video on the Korela Fortress Museum was realized . This panoramic video tour with interactive controls is designed for viewing in the Oculus Rift virtual reality headsets.
Interactive interaction allows you to select the route: the user, at certain points - forks, is given the opportunity to choose the desired continuation of the tour, or go back. Hovering on an element is performed by turning the head, which is tracked using the Oculus Rift. When you hold the sight on the selected element for several seconds, the element is activated and the next 360 ° video segment is launched. To improve the visibility of the interactive interaction when you hover the sight, the animation of filling the circular loading bar starts.
Map of interactive travels in the Korela Fortress Museum
At the aisles “forward” there is a professional guide, telling about the objects on the passable part of the path, the guide is no longer present on the aisles back in order not to duplicate the same information. If desired, the user can skip the video section by pressing a key on the keyboard or by using an interactive element.
The second form of interactive interaction is the ability to switch from a 360 ° video to a virtual three-dimensional reconstruction of the fortress. The reconstruction of the Korela fortress was carried out in the framework of the project “The Multimedia Information System“ Ancient Fortresses of the North-West ””. At certain points of the video tour, an element appears, activating which the user moves to a 3D reconstruction, with the possibility of free movement in the virtual reconstruction space using the Unity3D graphic engine. In the places of virtual reconstruction, coinciding with the real recorded static points-marks, in the reconstruction there are luminous portals. Entering such a portal, the user returns to the appropriate point of the video tour.
Conclusion
360 ° multimedia technology is a new and promising technology direction that allows you to create high-quality interactive content for virtual reality headsets in all areas of human activity (culture, art, education, science, tourism, advertising, industrial production, public administration, etc.) .
From the point of view of the entertainment industry and, especially, the creation of feature films in the 360 ° format, it is expected that a number of new approaches and methodologies of creative and technological processes will be developed that will allow the viewer to dive into the new virtual and interactive levels.
Authors of the article: Borisov Nikolay, Smolin Artem, Stolyarov Denis, Shcherbakov Pavel, Trushin Vasily
In the English version, the materials will be presented in May at the international conference "ArtsIT, Interactivity & Game Creation" (Esbjerg, Denmark). Here is the Russian, slightly adapted original.
Virtual reality and its application
Currently, virtual reality technologies and their applications are being intensively developed. Great opportunities are opened by the use of such technologies in culture and art , as they provide the effect of the presence of the viewer in the center of cultural events. One of the most promising and currently actively developed approaches to creating content for virtual reality systems is the 360 ° video technology, which allows you to create panoramic videos with varying degrees of interactivity, where the viewer controls the viewing angle of the video at will. This video can be viewed as in a virtual reality helmet (for example, in the Oculus Rift , Samsung Gear VR , HTC Viveand others), and with the help of a special application on the smartphone, while the picture changes in accordance with the turns of the user's head. You can view 360 ° video on the display of a personal computer. In the latter case, the user controls the view using the mouse or keyboard.
Watching video in 360 ° format in Samsung Gear VR headset
Many large companies (Facebook, Nokia, Samsung, Google, etc.) are currently developing cameras for shooting 360 ° video, virtual reality headsets for various smartphones and desktops, as well as various sound recording devices that provide the creation of surround (binaural) sound and allow you to realize a whole complex of 360 ° multimedia technologies.
The use of 360 ° video technology in education is especially important. Interactive video 360 ° can provide interaction with the end user, so this method can be widely used to train specialists in various fields where the development of practical experience requires a presence at the center of the event. The 360 ° video technology allows you to analyze the teamwork of staff during the performance of various tasks, which leads to an increase in the quality of the educational process.
A significant role is played by a 360 ° panoramic video to preserve and ensure access to cultural heritage. This technology provides a unique opportunity to see with the immersion effect not only closed for visiting archives of museums, but also reconstructed historical monuments that were destroyed by time or circumstances (three-dimensional virtual reconstruction). In addition, virtual reality is a unique opportunity for people with disabilities to travel and walk around the world.
Speaking about entertainment in the aspect of technology video 360 ° - viewers can visit the largest cultural festivals, find themselves on stage with their favorite musicians or watch a theatrical performance from the royal box.
The ITMO University Design and Multimedia Center is implementing a project for the development and application of 360 ° video technologies in culture and art Video360Production.com . Within the framework of this project, it was possible to achieve certain results necessary for obtaining high-quality content, starting with the development of original video recording devices 360 ° and sound and ending with the development of software for solving a number of tasks in this area.
Multimedia technologies 360 °
Special cameras are currently being developed with the help of which one could shoot video in 360 ° video format. However, with the help of special tools and software, you can use existing cameras to shoot 360 ° video.
A striking example of this approach is the wholea number of projects offering special rigs for GoPro HERO 4 cameras. The number of cameras in the riga is from 6 to 14 (stereo versions), depending on the tasks. After shooting, video streams are stitched into the general video sphere using separately supplied programs.
In addition, several large IT companies and electronics manufacturers announced their own fully automated solutions for shooting content for virtual reality - Google Jump + GoPro Odyssey , Samsung Beyond , Nokia Ozo .
GoPro Odyssey, Nokia OZO and Samsung Project Beyond
We described our personal impressions of the Nokia OZO camera here .
The 360 ° video is a new and progressively developing direction that still does not have standards of direction, which provides a wide scope for creativity and the development of new ideas and approaches. The effect of full immersion is achieved by a camera with multiple lenses, which allows you to shoot everything that happens around. However, this entails many difficulties, because many laws of direction do not apply when shooting in 360 ° video format. The main cardinal difference in viewing 360 ° video in virtual reality headsets is the fact that the viewer has the opportunity to watch content around himself, which means that the director will have to provide the viewer with the ability to dynamically select the viewing angle and not focus on a specific part of the screen.
The solution to this problem is the detailed reasoning of the scenario from the point of view of a paradigm shift - the departure to 3D video space. Each shooting requires both standard preparation (setting up the light, the most advantageous points for the camera, scenario and positioning of the scene), and non-standard approaches (special and often unique for each camera attachment, direction taking into account the features of 360 ° video recording, correct camera orientation, surround sound recording). When recording concerts, to achieve greater realism, the sound from the camera or from the stereo microphone is often combined with the sound from the director's console.
Representation of cultural events using 360 ° multimedia technology
Video360production.com collection of video footage for virtual reality headsets includes more than fifty different videos: concerts and music festivals, tours of historical places, tours in museums, theatrical performances, various cultural events and technological processes.
Group "Bravo", "Love does not burn" in the format of panoramic video
Grouping "Leningrad" in the format of panoramic video
The technology of laying asphalt "Ecohiter", panoramic promotional video
When recording concerts, it is necessary to transmit in high quality, both image and sound. Moreover, it is necessary to think about the location of the camera, since when recording to a 360 ° camera, such concepts as “front” and “background” become irrelevant. In addition, sometimes the scene was simultaneously up to twenty people, which complicated the installation of the camera. The footage allowed the musicians to look at themselves from the outside, to assess the location of the musicians during the concert, as well as to see the public reaction to the performance. The spectators, in turn, with the help of this survey were able to examine in detail the play of each musician and immerse themselves in the atmosphere created by the orchestra or group.
A concert of the Picnic group in the format of a panoramic video.
During the shooting of the famous Russian group Picnic, the following difficulties arose: the concert was held in a dark room of the concert hall under the bright light of spotlights. These conditions are difficult for ordinary cameras, because even at maximum resolution, the transmission of realistic color depth is quite a challenge.
Video recordings of concerts in the 360 ° format have a huge advantage over standard video recordings, as the viewer feels the effect of being present in the stage space and has complete autonomy to decide for himself where to look at the moment — whether to watch the guitarist or look at the soloist, or maybe see concert "eyes" of musicians, turning his back to the stage, and watch the lively reaction of the public.
To achieve the effect of complete immersion, not only a high-quality picture is necessary, but also a “live” sound. For musicians and frequent visitors to concerts, it is not a secret that the “technical” sound on the stage is far from what is heard in the hall. The sound engineers of the project combined the sound from the stereo microphone with the studio recording. As a result, the sound remained high quality, but it retained its “vividness”: while watching the recording, you can hear how the audience reacts and sings.
In addition to musical groups, the performers of the project video360production.com cooperates with the largest St. Petersburg theaters, where a number of performances and musical performances were shot. A unique opportunity to provide shooting historical sites in cities and excursions in museums. In the Museum of Anthropology and Ethnography. Peter the Great (Kunstkamera) in a 360 ° video format, before the reconstruction began, a large academic ( Gottorp ) globe was filmed - a diplomatic gift to Tsar Peter I from the Duke of Golstinsky Karl Friedrich during the Northern War.
Shooting a large academic (Gottorp) globe in 360 ° video format
The implementation of a joint project of the ITMO University and the State Hermitage Museum “Points of Presence” has begun, in which it is planned to present events in various areas of the museum using panoramic video filming technologies.
The shooting of the Peacock Clock video in the State Hermitage in the format of 360 ° video.
A virtual walk along the rivers and canals of St. Petersburg is also of heightened interest among YouTube channel visitors created by the authors of the project.
Particular attention was paid to filming a 360 ° video of various events in the Solovetsky Monastery. The Solovetsky Monastery played an exceptional role in the history and culture of Russia. Originating in the XV century. as a place of desertification of spiritual followers of the "father of Russian monasticism" Sergius of Radonezh, in a short time he became the spiritual, cultural, political and economic capital of the North of Russia. By the works of the Solovetsky monks, who gave the Russian Church a whole galaxy of holy ascetics, the Christian faith spread to the endless expanses of the Russian Polar region.
The activity of the Solovetsky Monastery decisively contributed to the influx of the Russian population to the White Sea region. The marine monastery had a decisive impact on the development of the economy and the marine part of Pomerania. This was a prerequisite for the development of the Arctic Ocean by Russian sailors and the transformation of Russia into a great Arctic state.
The geographical remoteness of the Solovetsky Monastery makes it difficult for many to take a personal part in the events taking place there. Shooting such events in the technology of video 360 ° and providing access to them via the Internet allows, when using virtual reality gadgets, to feel like a participant in such events.
As a result of the work done, more than 10 scenes were filmed on the Solovetsky Archipelago: Excursion to the Ascension Church on Sekirnaya Hill, Service in the temple of the Solovetsky Monastery, Christmas Liturgy, All-night Vigil, Children's Christmas Festival in the Solovetsky Monastery, Monastic caroling, various kinds of the monastery and the Big Solovetsky Island.
Service in the temple of the Solovki monastery in 360 ° video format
Excursion to the Church of the Ascension on Sekirna Gora (Solovetsky Islands) in 360 ° video format
Interactivity in 360 ° video format
Despite the fact that viewing 360 ° 3D video in various virtual reality headsets provides quality virtual immersion in video content; the next step is the possibility of introducing various interactive elements into the video format of 360 °.
Such elements can be:
• Active tags inside the virtual space for movement along various trajectories previously captured in 360 ° video technology;
• Introduction to 360 ° video of various additional content (images, video, hyperlinks, etc.) - the function “picture in picture”;
• Transition from video in 360 ° video format to simulated 3D space of reconstructed reality.
On the basis of the Unity3D system in the Center for Design and Multimedia of ITMO University an interactive tour in the format of 360 ° video on the Korela Fortress Museum was realized . This panoramic video tour with interactive controls is designed for viewing in the Oculus Rift virtual reality headsets.
Interactive interaction allows you to select the route: the user, at certain points - forks, is given the opportunity to choose the desired continuation of the tour, or go back. Hovering on an element is performed by turning the head, which is tracked using the Oculus Rift. When you hold the sight on the selected element for several seconds, the element is activated and the next 360 ° video segment is launched. To improve the visibility of the interactive interaction when you hover the sight, the animation of filling the circular loading bar starts.
Map of interactive travels in the Korela Fortress Museum
At the aisles “forward” there is a professional guide, telling about the objects on the passable part of the path, the guide is no longer present on the aisles back in order not to duplicate the same information. If desired, the user can skip the video section by pressing a key on the keyboard or by using an interactive element.
The second form of interactive interaction is the ability to switch from a 360 ° video to a virtual three-dimensional reconstruction of the fortress. The reconstruction of the Korela fortress was carried out in the framework of the project “The Multimedia Information System“ Ancient Fortresses of the North-West ””. At certain points of the video tour, an element appears, activating which the user moves to a 3D reconstruction, with the possibility of free movement in the virtual reconstruction space using the Unity3D graphic engine. In the places of virtual reconstruction, coinciding with the real recorded static points-marks, in the reconstruction there are luminous portals. Entering such a portal, the user returns to the appropriate point of the video tour.
Conclusion
360 ° multimedia technology is a new and promising technology direction that allows you to create high-quality interactive content for virtual reality headsets in all areas of human activity (culture, art, education, science, tourism, advertising, industrial production, public administration, etc.) .
From the point of view of the entertainment industry and, especially, the creation of feature films in the 360 ° format, it is expected that a number of new approaches and methodologies of creative and technological processes will be developed that will allow the viewer to dive into the new virtual and interactive levels.
Authors of the article: Borisov Nikolay, Smolin Artem, Stolyarov Denis, Shcherbakov Pavel, Trushin Vasily