How to connect Java, Js and graphs with art, or the story of how to create an interactive theater

    This year in Estonia there is a series of theatrical performances, united in the series “History of the Century”. During the year, 22 theaters in the country presented to the public in their performances the centenary history of Estonia. During the draw, the Estonian Russian Theater was given the topic of the future of Estonia.


    Under the cut - how to combine technology and art.

    Each of us has an idea of ​​what the future could be: what we fear, what we dream of, what it may actually be. To avoid a performance that tells only a narrow circle of theater people who think about the future, we asked this question to those who will live in the future - children, Estonian youth aged 3-19 years. There were hundreds of respondents, and we are very grateful to them.



    Each answer received is unique, but, sorting the materials, we saw noticeable trends and patterns. From the children's answers loomed an incredible number of possible worlds of the future - and attractive, and terrible. But these worlds also had common features, on the basis of which the plot was built.

    The performance takes us to the year 2118. Estonia is under a protective dome. We have learned a lot, including life extension. The main characters - an elderly couple Linda and Timo. A hundred years ago, as children, they fantasized about the future in which they now live. This is one of those very children, thanks to whom this performance turned out. Heroes fall into different worlds, in different Estonia. During the performance, it is the spectators who, by their decision, influence what their 2118 year will be.



    The theater does not start with a hanger, but much earlier. After processing all the material, the main ways of development of the future were identified according to the children, from the information technology path to the ecological state. Since the future is not predetermined, but depends on the choice of each of us, the possible paths of development were divided into stories, united by the history of the main characters and their families.


    Artyom Gareev (director of the play) shows the main ideas on a model

    The idea of ​​scenery is a constructor, which in the hands of children can take any form and function. He also has the form of a graph. A graph is a special abstract mathematical object that allows you to describe and model many real-world phenomena: the road network, the organizational structure, the Internet, and various choices that people can make. Elections that lead on different lines of the plot.



    In addition to the dramatic task, there were considerable challenges from the technical side: stage lighting, a complex system of controllers that controlled hundreds of LEDs on the scenery, a variety of costumes for each future variant, skillfully performed by one of Estonia’s best theater artists Rosita Raud. The actors needed to change quickly after each vote, which was especially difficult when the intrigue continued until the very end of the vote and the next scene was unknown.

    All music is played live by the MODULSHTEIN project .



    To make it a little clearer, let's get into the context of the play a little.


    We ended up in a multi-cultural society where languages ​​and genders intermingled. Everything is allowed.
    A news release immerses us in the proposed circumstances.


    Ironically adds that Elena Solomina - leading on national television and is well known to many viewers.

    Multicultural society is sheer fun, the younger Semenova leads to get acquainted with the parents of a huge boyfriend dressed in arafatka and a tutu.

    After all, they go to a party.




    At the end of the scene, a vote opens, offering the choice of whether Estonia will take the proposed path of multicultural development.

    And if the answer is positive, then ...

    Utopia falls into dystopia and freed freed up with a knot. There are new bans, and now the special forces break into the party and sort people. Some are passed to the right, towards the future, others are stripped and sent to the left, where their fate is decided by the destruction machine. (based on articles by Nikolai Karaev and Boris Tukh)




    What technical solution did we need?


    In fact, it was another EMA actor (Est. Mama), artificial intelligence, which became a reality in the future. EMA leads the audience through the plot, manifests itself in various variants of the future, and also announces and launches the vote, counts and presents the results, moves along the subject lines. The hall could accommodate 600 people. And each viewer should have the opportunity to vote, and the system correctly calculate his voice.

    Possible options for the development of the plot, presented and modeled as a graph, formed a complex and branched structure.

    Utopia tends to flow into a dystopia, and the road to hell is paved with good intentions. The plots are interconnected by the logic of drama. Considering the number of options for the future that the children offered, a large number of branches came out. It is important to avoid repetitions - what was played should not be put to the vote again as a possible option. For example, if, after a reviewed Information and Technological Utopia, viewers confirm their choice of such a future, then we fall into a technological dystopia. If the audience rejects the proposed option, they must again make a choice between the proposed options - for example, the Ecological and Cosmic-Colonial future. Very often there were situations when the hall struggled, the votes were divided almost 50% to 50% and everything was decided by a difference of several votes.


    All this complexity required an appropriate structure, which is the graph.



    What other functional requirements were there for the technical solution for the performance? We needed:

    • The voting system, which collects votes during the voting period, shows the voting process and their results.
    • Automatic selection of actions - depending on the results of a vote, the opening of a new vote or the announcement of the beginning of the next storyline, in accordance with the logic of the graph.
    • The system should communicate with viewers, offer choices and announce results in natural language.
    • The system should be managed by an assistant director and display complete statistics on the progress of the vote.

    So we have 600 people who must vote in less than a minute. People should be able to cast a voice from their phone. The mobile application option disappeared immediately, because downloading the application is an extra effort. Nobody downloads apps if you are not Facebook or Google. Different models and phone types require different designs, which is unrealistic in the conditions of 1.5 months and one developer. Consequently, we need a Web application for a mobile phone.



    Another Web application is required for system administration, startup, activation of states and other technical issues.

    And finally, the most important thing is how the EMA should interact with the hall.



    And here it is necessary to introduce the concept of "super", which was new and for me. Super is a screen that goes down and closes the stage.

    Lifting curtain
    Also known as “guillotine curtain” or “German” curtain. More often used in musical theaters. The curtain is a whole canvas, straight or pleated, fixed on the upper edge to the carrier beam. The lifting of the curtain occurs simultaneously with the lifting of the carrier beam. Such a curtain requires a great height of space above the stage.
    (Wikipedia)


    The curtain is reflective and allows you to display various projections, such as news, about which will be below, or the EMA screen, which displays information related to voting, its results and the choice made.



    So, we have a complicated graph, a voting system and three FrontEnd systems that must interact with the audience and the administrator (an incredible assistant director Svetlana Shushina).

    Decision


    So let's start with the graph. The data structure, in this case variants of the history of Estonia, must be stored somewhere. Standard relational databases are not suitable for graphing for several reasons. For us, the most essential was the flexibility of the stored data, transitions, links to audio files depending on the choice, and so on.

    The definitive choice was Neo4j, the world's leading graph database system. 7 of the top 10 technology companies in the world use it. This is a very mature technology. Speed, convenience and excellent community leave no other choice in this kind of solutions.
    We modeled the plot, built by dramatists, in the form of a graph on Neo4j. And here is a piece of plot twists and turns, as it looks from the inside Neo4j.

    One of the advantages that Neo4j gave was flexibility and the lack of a scheme, which allowed to change the plot lines without any serious consequences, add new audio options, link various novels to each other. This is the native representation of the graph in the native environment.



    In terms of server side there were some doubts - Python vs Java. The choice leaned in favor of Java. The reasons are speed, less risk of breaking something with clear requirements and straight arms, and finally Spring Boot, which supports Neo4j out of the box.



    The server part is implemented on Spring Boot. In fact, the server collects votes, switches from “super” to the following states when the voting is completed, accepts commands from the admin, gives statistics and so on. Plus gives all the options FrontEnd-am, and also registers the audience.



    Now begins the most non-trivial part of the task. The fact is that according to the idea of ​​the component that is displayed on the "super" should talk with the audience: prepare to vote, suggest options, start and end voting, announce the results and, depending on the choice, suggest other options, or go to the choice and tell the introduction to the next scene. Thus, the audio must be dynamic and prepared on the fly. Given that the number of options is huge, and the length of each segment of audio is different, synchronization is important (for example, voting should open only after the announcement “Voting is open”) and this was not a completely trivial task.

    The system has two control centers.

    The first is the server. It stores the current state of the plot, collects votes, gives statistics data, controls the ability to give only one vote from one person.
    The second is “super”, which turns the voting on and off, shows the voting process, and so on.

    "Super" is implemented on Vue. Initially, we wanted to call audio with the Google Speech API, but we could not guarantee the absence of network delays. Therefore, another option was chosen - we recorded the response of the Google Speech API piece by piece.

    In total, we got 55 pieces of audio. Information about which audio files should be included depending on the current state, we stored on the graph. Since we are dealing with JavaScript, we stringed audio files and command calls on each other, like beads. Thus, we did not depend on the fact that audio files of various lengths, and got a fairly universal solution.

    Initially, the mobile client (web page) was written on Vue, but we got a partial incompatibility on some phone models. We tried the babel transpilers, but because of the heap of settings, the volume of the page grew, and although voting began to work on some telephones, it broke down on others. As a result, the mobile client was rewritten on Vanilla JS.

    In fact, it turned out to be a distributed system.

    The final volume of the system (jar file) was 146 megabytes. The telecommunications company Telia provided us with a virtual server at a symbolic price of 1 euro per month, for which she is very grateful.

    The pace of work before the premiere all accelerated. At the premiere of the expected president. The last bug was detected in half an hour, and there was no time for testing, so the first performances went with parallel testing locally on the machine behind the scenes, so that in case of an error it was possible to draw up a plan of action.




    Of the interesting moments, we can highlight the fact that we showed the names of the last voters on the screen and used the animation of the Vue templates for this. At the first submission, we were given a killer character, which broke the animation and display of Vue templates. Subsequently, we abandoned this solution and switched to the good old jQuery.

    But a piece of voting taken during the performance.


    The performance received a huge number of positive reviews in the press
    https://rus.postimees.ee/6141942/budet-ne-budet-net-nichego-opasnee-budushchego
    https: // rustime- ee/ 6147704/ nashe- zavtra- v -zerkalah-utopii-i-antiutopii


    At the Drama Festival in Tartu, the applause lasted almost 5 minutes.

    It was an incredible adventure due to the crazy energy and dedication of the people involved in the project. A huge bow to all the people who participated in the creation of this show.

    Director and Set Designer - Artyom Gareev
    Composer, producer of the project - Alexander Zhedelev
    Visual - Alena Movko
    Costume Designer - Rosita Raud
    Lighting Designer - Anton Andreiuk
    Choreographer - Olga Privis
    Playwrights: Karin Lamson, Mari-Liis Lille, Elena Chicherina and Laura Kalle
    Video - Nikolai Alkhazov
    Programming and creating an application - Alexander Tavgen, Anna Agafonova
    Technical solution ice lighting - Aleksander Sprohgis
    Animation - Martin Yakush
    Music band - Modulshtein
    Marten Altrov - clarinet, bass clarinet
    Alexey Semenihin- work with samples, effects
    Alexander Zhedelov - guitar, synthesizers, ableton
    singing teacher - Anna Dydyna
    assistant director Svetlana Shushi
    The cast:
    Natalia Dimchenko Alexander Zhilenko Daniel Sandberg, DMI ry Kordas, Catherine Kordas, Alexander Kuchmezov Victor Marvin, Natalia Murin, Elena Tarasenko, Edward Tee
    Sergey Furmanyuk, Leonid Shvetsov, Elena Yakovleva
    Studio Russian Theater: Denis Volkov, Pauline Grinyova Nina Zagvozdkin Anastasia Kalyada Natasha Christensen, Anastasia Masalova Sandra Minosyan Sofia Mikhalev, Catherine Magi, Michael Pashchuk, Catherine Selyugina Christina Sorokina, Sofia Strömberg , Ilya Sutt.

    Also popular now: