Is there any powder in the flasks? Hackathon Radio Canada 2018 (Part Three - Go To Start! Attention! March!)

    I present to you the third part of my short story.

    Having received a positive assessment of the first and second parts, I did not want to force readers to wait too long, but life and reality are making adjustments.

    image

    2 days before the start of the hackathon, Radio Canada sent a letter informing us that our team had a mentor - Patrick Lévesque . For me, this was somewhat unexpected, although, probably, something was written about this on the site and was said during the preliminary presentation. Apparently, I missed it by my ears. One way or another, this added confidence that we would have someone to ask for help in case of any questions.

    Tuesday, Wednesday, Thursday of the working week flew by rapidly as always. The anticipation of the adventure grew. And by Friday reached its peak. My wife was warned that on the weekend, starting on Friday, I would fall out of life and disappear from the radar.

    So after work on Friday, I went back to the House of Radio in a fighting mood.

    The first day. Friday evening. Time to throw stones


    Friday schedule
    You can see the original here . But the question is: who needs it? I think it is in Russian and few people are interested. But on the other hand, someone is reading and plusing my story, what if these very little things are my highlight?

    Friday, March 23,

    5:00 p.m. Arrival and check-in (Radio-Canada Dining Room closes at 7 p.m.).
    18:00 Greeting
    18:15 Presentation of mentors, assistants, teams and tips
    19:00 Workshop on preparing a presentation of prototypes with Matthieu Dugal and Chloe Sondervorst
    19:30 Break
    19:50 On your choice:
    1. Workshop
    Presentation of Microsoft Services
    Presentation -Radio Canada Services
    2. Getting Started
    22:00 End of the first day

    The mention of the dining room was very appropriate. Firstly, it reminded me that I would be after an empty day with an empty stomach, and something to eat before the start of this action would not even hurt. The asterisk even offered to bring me food, because she came to the hackathon not from college, as usual, but from home, quite well-fed and full of energy. But as always, I outwitted myself. I thought that we would have time, desire and the opportunity to order pizza. But in reality, it turned out that I was the only one who came on an empty stomach in our team. So I did go to the dining room and took myself some kind of hamburger, right after registration.

    I arrived at about 17:45.

    Before entering the hall where the event was held, it was necessary to register and get a t-shirt of a hackathon participant and a small souvenir in the form of a bottle for water. The participants were given white t-shirts, and the organizers and mentors were black. My attempt to get hold of a black T-shirt was unsuccessful. Well, okay, white is also nothing.



    When I entered the hall, Zvezdochka and Plato had already taken a fancy and occupied one of the tables reserved for the teams. We were engaged in connecting laptops to power and the Internet.

    I also prepared my laptop, and then used a couple of minutes left before the presentation to go to the dining room and have a bite to eat.

    By the time I returned to the hall, Mercury had already joined the team. There was no chaise, and when he appeared it was unknown.

    With a slight lag behind the schedule, at about 18:10, we listened to a welcome speech from Maxime St-Pierre . He reiterated that Radio Canada had high hopes for the event, as last year’s experience had good practical results. He also mentioned that the hackathon theme: AI, is very relevant today. And confirmation of this, for example, the scandal with Cambridge Analytica that happened literally this week .

    You can view the full recording of this evening’s broadcast:


    Maxim did not drag out his speech and handed the baton to the evening host Matthieu Dugal .

    Matthew, as I understand it, is a very famous television presenter on Canadian television. He runs several technology programs. This played a positive role in organizing both this evening and the entire hackathon. On the one hand, his experience in television format helped a lot in terms of time management. All points of the program, despite the delay in the beginning, ultimately completely fit into the program announced in advance. On the other hand, he did not have any difficulties pronouncing technical terms, which I think would be the case if the presenter was not familiar with the technical subject.

    Over the years, I have more and more respect for people and organizations that know how to “answer” for their words. Despite the availability of more and more tools available to everyone and the universally declared “professionalism”, so far, I very often meet people and situations where the organizers are not able to control their own events. In this regard, Radio Canada were on top.

    At first, Matthew once again spoke about all the requirements of the hackathon and the criteria for evaluating prototypes. I will not list them again. You can re-read them in the first and second parts.

    Prototype Evaluation Criteria
    It seems that I did not describe the criteria for evaluating prototypes in the first two parts. Strange, I was sure I mentioned them.

    In French
    Les critères d'évaluation sont les suivants:

    Le prototype tient compte des critères obligatoires du règlement (API et Azure) / 20 points
    Originalité / 35 points
    Excellence de la technique / 20 points
    Impact / potentiel / durabilité / 20 points
    Présentation / 5 points

    • The prototype must meet the declared requirements listed earlier - 20 points
    • Originality of the idea - 35 points
    • Technical performance - 20 points
    • Possible impact on business / commercial potential - 20 points
    • Presentation of the prototype - 5 points


    He also once again announced the list of prizes for the winners.

    Prize List
    1st place - XBox One + $ 1000
    2nd place - Google Home Max + $ 250
    3rd place - $ 500
    And two prizes (Xbox One S1), which will be drawn among the remaining participants by drawing.

    (as far as we understand the prizes each team member received)

    After that, Matthew introduced all the mentors and assistants
    and each team member by name .

    Despite the fact that 31 teams were represented on the screen. My team and I were still under the illusion that half of these teams would either not participate at all, or could not cope with prototyping, or they would be significantly weaker than our team. At that time, I did not even count the number of teams. I was too busy setting up access to Azure and generally watched and listened to everything just by the edge of my ear. Although the view of the people in the hall a little worried me. In the sense that I assumed that most of the participants will be university students. And my eyes told me that people with a beard and glasses are unlikely to be students. That is, visually the composition of the participants was quite mature.

    After adjusting the schedule a little, Matthew announced a break for 20 minutes. It was very helpful, everyone had to exchange opinions, walk, stretch themselves, refresh themselves.

    During the break, Zvezdochka asked once again to explain to Mercury the architecture of our project and the role of each team member. Which I did on paper.

    A little more beautiful and more structural than then on paper
    Roughly something like:



    1. WEB page / Widget - HTML + JS


    functionality : the visible part of the application organized in the form of HTML and the search input field of the processed JS.
    Performers : Asterisk - HTML, Mercury - JS marginal
    notes : I suggested 3 implementation options for this part: a separate page (the simplest frontal solution), a field embedded in the Radio Canada page (I thought the guys could reuse the page structure and CSS classes for drawing search results in the form of blocks / widgets, so as not to waste time creating your own) and the third option is the same as the second one, I proposed to make the search field field in the form of Chrome Extension, because I knew what to make the simplest extensionit’s not difficult for a browser. And installing such a plugin allows you to access any page, including the page we need with the news feed.



    2. PHP, MySQL - Search API backend


    functionality : the simplest search api, according to our plan, it should accept a request from the search line on the page, search the database and return JSON formatted search results.
    artist : Your humble servant marginal
    notes : since the last project in which I participated in my main work was done at Laravel, and since this framework is a trendsetter today, I decided that I would use Laravel . This should not have brought any special surprises, but for my part I will learn some features of Laravel when creating a new project on it, especially not in the mode, so to speak, of the site, but in the API mode of the service.

    3. MySQL


    functionality : database, storage for content and analysis results marginal
    notes: Since there were no strict requirements on which database to use, Plato asked if it would be better to use MongoDB. For MongoDB, they say JSON is almost native and the type should be simpler. Honestly, I can’t answer, and now it’s more correct to make such decisions on non-relational databases. From what I read on this topic, I conclude that if a fairly simple selection is required for previously known fields, and, in our case, in fact, for a single table, then relational databases are still the best solution. (Correct me if I'm wrong). But my choice of MySQL was not due to architectural preference, but to pragmatic considerations: I just know how to use it. For Plato, both bases were something new and, in any case, an experiment, so he easily agreed with me.

    4. Python App


    functionality : (Radio Canada -> Python App -> Azure Cognitive Services -> Python App -> MySQL) This part is essentially the most important in terms of hackathon requirements. You need to get the content through / from the Radio Canada API, possibly process it, then send the content for analysis to some of the AI ​​services, get a response and put the content along with the analysis results into the database.
    contractor : Plato marginal
    notes : The entire burden of responsibility thus fell on Plato. He was a little worried about this. But as I already noted, he spent his homework and felt that he had all the knowledge necessary for the project. Plato, like the whole team, had no doubt that we would cope with the task.

    5. Azure Cognitive Services


    functionality : at the beginning of the hackathon, I did not know which of the services would be involved

    6. Radio Canada API


    functionality : at the time of the start of the hackathon, I did not know which of the 4 services would be used by Plato (read the description of the 4 API Radio Canada in Part 2). As in the case of AI services, the decision should have been made by Plato, based on the fact that it is easier and more convenient for him to implement.

    After the break, Matthieu Dugal and Chloe Sondervorst held a master class on preparing the presentation of prototypes. They drew the audience’s attention to the fact that, firstly, you need to understand that your prototype product is “met by clothes”, and therefore success and chances of winning will very much depend on the presentation of the material. The second thing the presenters turned our attention to is the limited time. On the performance of each team on the final day, 3 minutes are given for the performance itself and another 2 minutes for answers to the jury's questions, if any. 3 minutes is really not enough, so you need to focus your speech on the most vital moments and correctly place the emphasis in the speech.

    If we make a simple mathematical calculation, 31 command x (3 min + 2 min) = 155 min = 2 h 35 min. And then we add a couple of minutes to all kinds of hiccups, changing teams on the stage and so on, it becomes obvious that only listening to the teams will take 3 hours at best. But I repeat, at that time I still did not understand that 31 full-fledged teams were participating in the race . If you remember from our original plan, when we registered the team, we assumed that in reality there will be no more than 10 teams, of which perhaps a couple will fall off by themselves.

    Chloe sincerely tried for this short time to give a short course in public speaking and oratory. She told what are the main criteria by which you can evaluate your presentation: first impression, history, credibility / realism, simplicity, time (time management), emotionality, vision, passion / emotional passion.

    Concluding their part of the speech, the presenters once again drew attention to the stated criteria for evaluating prototypes.

    After the end of the master class, the participants had the opportunity to ask a few more questions. Some of these questions related to how functional the prototype should be, who will test it and how and whether to include a real demonstration in the presentation, or whether some static slides with screenshots or something like can be dispensed with.

    Answering one of these questions, Maxime St-Pierre clearly stated that the prototype should definitely be functional and it is best to include its demonstration in its final presentation.

    I want to draw your attention to this small touch. Because we based our expectations precisely on the assumption that the prototype must be ready and it must meet the
    declared criteria.

    After answering questions, we once again wished good luck. And the work began to boil! (approximately 20:00)

    A lot of text for fans to read
    Rather, as a work, a certain activity. For my part, I immediately explained to all members of our team that we should, as far as possible, conduct the development of our parts as independently as possible. At the same time, we must try to avoid blocking each other. According to our architecture, it turned out that the points of possible mutual blocking are transitions: JS <-> PHP, PHP <-> MySQL <-> Python. That is, my part in a certain way can be delayed / blocked by unavailability or problems with the database. And I, in turn, may turn out to be a blocking link for our front end.

    Therefore, the first step was to raise the database and check access to it from both PHP and Python.

    Plato, meanwhile, dealt with both APIs and determined which way we would move.

    Asterisk and Mercury went about their business. By the way, they pleasantly surprised me by the fact that they immediately raised the common repository on Github and really engaged in joint development from the first minutes, dynamically exchanging updates. By the way, I still haven’t uploaded my code anywhere, although several weeks have already passed. (I'm ashamed, honestly)

    I asked Zvezdochka to proceed from the fact that it will receive a certain JSON array with elements from the PHP service, each element will have the following minimum set of fields: video_id, title (title), category (some category to which the content belongs, maybe it will be: sport, politics, economics, maybe a musical style or a performer, or maybe just a Quebec region), body / description (article body), image / video (url to a video or picture). And accordingly, she must create an HTML layout for the output of these elements. I asked her that the layout should be designed for 6 elements in 2 categories.

    First Alpha, appearance, front end
    That's what our front-end team was about ready for the end of the day.

    HTML generated
    Card image cap
    Will need info from api

    Some quick example text to build on the card title and make up the bulk of the card's content.

    Read more ...
    Card image cap
    Will need info from api

    Some quick example text to build on the card title and make up the bulk of the card's content.

    Read more ...
    Card image cap
    Will need info from api

    Some quick example text to build on the card title and make up the bulk of the card's content.

    Read more ...
    Card image cap
    Will need info from api

    Some quick example text to build on the card title and make up the bulk of the card's content.

    Read more ...
    Card image cap
    Will need info from api

    Some quick example text to build on the card title and make up the bulk of the card's content.

    Read more ...
    Card image cap
    Will need info from api

    Some quick example text to build on the card title and make up the bulk of the card's content.

    Read more ...



    I started several VMs, one with Debian Jessie, the other with Ubuntu 16 and tried to install MySQL on them - it didn’t work. Either I was in a hurry or too tired, I don’t know why. Even the VMs themselves did not rise from me on the first try. Since I did not want to bother anyone until everything worked, I tried to start each VM in a separate security group. Azure spat and said that I was not supposed to (restricted). I still did not understand why it was not supposed to. Plato told me when I desperately complained to him about the stubbornness of Azure. Then it went a little easier. All the machines and services that I used after that, I created in only one security group and subnet, which was originally on our account, and naturally in the same region.

    On the other hand, what I liked about Azure VM is that, unlike AWS, I have a choice: I can have SSH access to the machine using a private key, or using the traditional “username” - “password” pair. I am aware that the AWS approach is probably safer. But in some situations (such as this one), when security issues are temporarily irrelevant, the account itself is temporary, the team members do not have sufficient qualifications, and there is simply no time to train them, in such cases it seems to me to exchange simple pairs: “ username ”-“ password ”is more efficient and quite secure.

    So, I tried to raise MySQL on two VMs and open port 3306 to the outside so that it would be easier to test and connect to the server. At the same time, I raised 2 or even 3 MySQL databases as an Azure service, hoping that the service should be easier and more convenient to use.

    Meanwhile, Olivier Fortin came up to the Star, as you remember, this is a specialist in Web Accesibility. He asked how our HTML code was up to standard. The asterisk reacted vividly to his question and showed everything that she had at that time. Naturally, the code was far from being considered valid for WCAG 2.0. But for us, this standard was something completely abstract and not the main priority. Olivier suggested exactly what you should pay attention to and recommended installing utilities that people with disabilities use to check how convenient it is to use our product using these utilities.

    By 22:00 when we were asked to round off I was in the middle of nothing. MySQL did not answer me anywhere and in any way. The mood was not very. And I was afraid that tomorrow I would really become a block for everyone. But thank God I'm not a boy. I calmly reasoned that Friday evening, the end of the work week, a lot of information, a slightly empty stomach, all of this together could be the cause of my failures. And following the Russian proverb: morning is wiser than the evening, it is worth going home and getting enough sleep, and tomorrow you will come back with fresh strength and a fresh head.

    Oh, by the way, I forgot a little, it seems Phaeton appeared at 20:30 or 21:00, I can’t remember, because by this time I was completely immersed in dancing with a tambourine experimenting with Azure.

    Second day. Saturday. Flour of creativity


    Saturday schedule
    Saturday, March 24

    9:00 Arrival, breakfast, continued work in teams
    12:00 Lunch, continued work in teams
    18:00 Dinner, continued work in teams
    22:00 End of the second day of the competition

    As you can see, the second day from the point of view of the schedule was the most concise.

    Work, work and work again.

    Despite all my impatience on Saturday morning, I came later than everyone else. Well, not all, but the fourth of our team by 9:30. I needed to bring my youngest son to a Russian school.

    Why do we take children to a Russian school
    Since we want the child not to completely lose his cultural connection with us and with his grandparents, aunts and uncles, sisters and brothers, who live far from us geographically, but are always close to us, we take the child every Saturday to a Russian school. By the way, it is the largest of the Russian schools located outside of Russia.

    We know many examples when children coming to Canada at a fairly young age and / or born here, after how many years they forget their native language. The fact is that many parents who came to Canada, in my opinion, focus too much on integration. I can understand them. Indeed, why was it necessary to move if not to integrate into the local community? And trying for their children they begin everything possible and in the family circle speak English or French. And children are built in so that they quickly learn a new language. The smaller the child, the faster he switches. And then the kindergarten and school do their job.

    In addition, additional classes on Saturdays or Sundays require additional effort, money, time, attention. Someone does not have enough money, someone has time, someone just gets tired of telling and proving to a child why he needs it. And the children grow up here, let's call it "freedom-loving." And believe me, every Saturday in the morning to take away and pick up a child 20 km in the evening, this is an additional burden that annoys and breaks many plans for a rare weekend.

    In addition to studying the Russian language, literature and geography, it is believed that this school (as well as several other Russian schools in the region) provide a good additional base in mathematics and physics, which allows students to better learn the original Canadian curriculum for primary education in public schools.

    Asterisk, by the way, also attended this school until admission to college. And she kept a very friendly relationship with many of her classmates from there. They are still regularly, albeit not very often, by virtue of employment, get involved and spend time together.

    I greeted my colleagues and prepared my workplace for work. Then Zvezdochka took me to an impromptu buffet where I could get coffee, juice and a muffin for a breakfast.

    Along the way, she shared her plans for the day. She said that work with Mercury is in full swing and I don’t really have to worry if they have questions or problems, they will let you know. We are back at our table.

    Plato informed me that he had decided on the direction of movement and was completely immersed in work. He also said in passing that MySQL was working. I was surprised, but did not delve into what exactly and how it works. It was a little embarrassing what earned Plato, what I had to prepare.

    It should also be mentioned that our mentor Patrick Lévesque came up to us twice the night before and on Saturday to make sure that everything was moving normally and we did not need his help. It was pleasant and strengthened us in our self-confidence.

    In the meantime, I asked Zvezdochka and Mercury to show where they were moving in order to understand how far they are from the possible blocking point, when we need to connect all our parts together. The star said that they are working on the Chrome Extension, which they easily managed to run in the simplest version of the tutorial, and now she understands how this browser extension can interact with the Radio Canada page.

    Mercury said that he is working on a dynamic input field so that when you press the next key, a request is sent to the backend. Thus, it turned out that right now, Mercury will need my backend service in order to move on not blindly.

    Leaving them to continue to work in the right and already slightly more definite direction, I went over to talk with Plato.

    Plato told me a rather large and important piece of information.

    Where are we going with Piglet?
    From Radio Canada, we will use the Lineup public news API .

    As an AI, the T ext Analytics service will be involved . Accordingly, after analysis, in addition to Radio Canada content, we will have two additional database parameters / fields: SENTIMENT (emotional assessment of the text: from 0 to 1, where 0 is negative text and 1 is positive), KEY PHRASES (keywords and phrases). I simply took this information into account and tried to analyze on the go what and how it changes in our project and how these additional fields can be used.

    To my question how Plato managed to raise the database, he said: so you raised it yesterday. It turned out that yesterday, during my experiments, I regularly dropped the name of the VM and the services that I tried to run, along with user credentials and passwords to Plato in Skype. I did it so to speak “in reserve”, so as not to look for a long time later, when I decide which service we will use. Plato, for his part, tested one of them and the service answered him safely, i.e. The database responded like a living one. I clarified with Plato which service or VM he was referring to and which utility. It turned out that since the database world is almost new to Plato, he simply knocked on the specified service from Python following some tutorial found on the Internet.

    On the one hand, it pleased me. It turned out that I still lifted the service. On the other hand, it was not clear why I couldn’t get connected to it yesterday from my favorite PHPStorm. The explanation was simple, Plato connected to the service from the inside, so to speak, while inside the Azure network and in the same security group on an already running VM with its Python. And last night I tried to set up and test public access for everyone. That is, that goal, as regards public access, I still did not achieve, but the database was available for work, and this is important. So, the option that worked for us was MySQL as a service from Azure. The rest of the VMs raised the day before, I simply did not use and did not extinguish until the end of the hackathon, simply because there was no time.

    Tick, my partners are all working, the clock is already 10 with a tail, and I still don’t even have a dev server with PHP.

    How the server rose with PHP
    This time for some reason everything turned out right the first time.

    Raising the new Debian Jessie VM. It takes a couple of minutes (about the same as it takes for non-AWS). I am successfully connecting via SSH to my old friend Putty, it pleases. Installing PHP 7.2, like I got up. Install Composer, the flight is normal. Install composer global require "laravel/installer", like there are no errors. Following the official installation guide , I create a Laravel project laravel new api. Everything went smoothly. Just a couple of steps: install Nginx, change the default config to point to the folder /publicin the folder with my Laravel project and call php-fpm for php files. In Azure, I open access to port 8080 for my VM with PHP.

    To guarantee, I tweaked the default welocome.blade.php to be sure that I see exactly my page and my project.

    Hackathon 2018 Radio Canada


    I check in the browser for the public IP of my virtual machine and voila!



    First goal: not to block the work of the guys on the front end. You need to create a simple controller and give them a pseudo answer, even if static. But at least they can already work out the interaction.

    The first iteration of PHP
    Create a new route in routes/web.php

    Route::get('/search', 'SearchController@index');
    

    and a new controller app/Http/Controllers/SearchController.phpwith the only index method / action yet

        public function index()
        {
            return response()->json([
            ['video_id'=>'1', 'title'=>'title 1'],
                ['video_id'=>'2', 'title'=>'title 2'],
                ['video_id'=>'3', 'title'=>'title 3'],
        ]);
    

    I check in my browser. It seems to work.

    [{"video_id":"1","title":"title 1"},{"video_id":"2","title":"title 2"},{"video_id":"3","title":"title 3"}]


    It was already some kind of progress, and I felt that like other team members that I joined the work. The anxiety that I would turn out to be a weak link has receded.

    The asterisk by this time began to adapt the code for Web Accesibility. To begin with, she installed NVDA from last night to be able to test her changes. During this day, she really became, if not an expert, then a good specialist in Web Accesibility, and I think that she, like myself, understood what the term “site semantics” means. When semantics becomes not just something theoretical, but has a very important practical value.

    For comparison, here is the final HTML that was presented in the final version:

    HTML - final version

    Région - Saguenay - Lac-Saint-Jean - Sports

    Région - Estrie - Sports

    Région - Bas-Saint-Laurent - À ne pas manquer



    If you compare with the original version, you will see that some tags
    were replaced by more semantically understandable
    и
    and for tags
     и 
    respectively added altand titleattributes. Prior to this hackathon, I attributed such modifications for the most part to SEO and considered this an excess, with a somewhat obscure application purpose. Now I think the applied purpose of such practices will remain in my perception of HTML code for a long time: these changes really help programs like NVDA to “read” a screen or web page in a more understandable, let's say directly, human language.

    The next step in my plan was to create a service for Zvezdochka, which will replace its static data with my static data, which I will subsequently replace with dynamic data from the database. I added another way to the Laravel config

    Route::get('/feed', 'SearchController@feed');
    

    And accordingly, another method in my SearchController

        public function feed()
        {
            return response()->json([
                ['video_id'=>'1', 'title'=>'title 1', 'description'=>'Description 1'],
                ['video_id'=>'2', 'title'=>'title 2', 'description'=>'Description 2'],
                ['video_id'=>'3', 'title'=>'title 3', 'description'=>'Description 3']
            ]);
        }
    

    First gluing front end - back end
    It was already the beginning of the 12th. I asked Zvezdochka to try replacing my data template with a call to my template. At this time, she was mostly busy modifying HTML and refining CSS. I want to note that after talking with Olivier, she refused to use bootstrap.css. I was a little disagree with this decision, but the lack of time did not allow me to delve into the discussion. I decided everyone makes his piece in the way that he understands. So the asterisk told me that the call to my service is not working! I double-checked several times in my browser - my service gave json fine.

    At first I got into the JS code, not really trusting the Asterisk. Moreover, she did not claim to be 100% knowledge of JS.

    I was a little discouraged to find something like this code:

    function parseJson()
    {
    	xml = new XMLHttpRequest();
    	xml.onreadystatechange = getData;
    	xml.open("GET","http://ХХХ.ХХ.ХХХ.ХХ/feed/");
    	xml.send();
    }
    function getData()
    {
    	if(xml.readyState == XMLHttpRequest.DONE && xml.status == 200)
    	{
    		console.log(xml.responseText);
    		jsonArray = JSON.parse(xml.responseText);
    		fillInCards();
    	} else {
    		console.log("There was a problem with connection");
    	}
    }
    

    I am not special in JS. But what I'm used to working with, if we are talking about AJAX, then this is mandatory jQuery or its analogues. I don’t remember why I formed such an opinion, but I am somehow internally against using raw JS for AJAX calls. Probably, due to my limited knowledge, I rely on libraries that by default will do some actions for me that I may forget or not know about: open / close a connection, handle a non-standard event, call some kind of handler, check / validate data before shipping and / or upon receipt.

    For me, a code of the form would be more familiar:

    $.ajax({
      url: "http://ХХХ.ХХ.ХХХ.ХХ/feed/",
      cache: false,
      success: function(data) {
          getData(data); 
        }
    });
    

    But what to do, Zvezdochka insisted that her code was verified in many laboratory works and projects that they had not previously done in college and should work. I had to believe her. Moreover, after a little investigation, I found



    that it seemed to me not quite logical. That is, the ajax request received a response with the status of 200 from my service, but did not receive the response body and returned an error to the JS console: No 'Access-Control-Allow-Origin' header is present on the requested resource. For about 5 minutes I was stupid trying to realize what was happening. Then it dawned on me that these were probably the default Laravel security settings or rather even the Chrome settings themselves, which by default allow AJAX requests only for their domain / host.

    With a little googling, I decided that I could just return the header 'Access-Control-Allow-Origin' along with the data, which should solve the problem. Having experimented a bit with my method, I got the look:

    public function feed()
    {
      return response()->json([
        ['video_id'=>'1', 'title'=>'title 1', 'description'=>'Description 1'],
        ['video_id'=>'2', 'title'=>'title 2', 'description'=>'Description 2'],
        ['video_id'=>'3', 'title'=>'title 3', 'description'=>'Description 3']
        ],200)->header('Access-Control-Allow-Origin',"*");;
    }
    

    After that, JS kind of stopped cursing. It seemed that the front-end-back-end ligament was taking shape.

    It's time for a second breakfast or first dinner, as you please, in the local place just lunch. The first to go have a meal is Zvyozdochka with Mercury. And Plato and I, briefly breaking away from the code, discussed the current progress and structure of the database table. Plato once again confirmed that the code for receiving news and sending them for analysis to the AI ​​is almost 90% ready.

    Unfortunately, Plato created the repository after the hackathon, so it is not possible to follow the development of his script over time. I will insert the entire script near the end of this part. In the meantime, I’ll just say that at the time before lunch, Plato said that I can expect about 100-120 entries in the database to appear soon after lunch. For the time being, it was quite suitable for me, it was felt that the development was precisely moving along all fronts as planned.
    And with a calm soul, we also went for a snack.

    What is the feeding of mental workers
    The food was hearty but weird. There was some type of porridge quinoa, но она была не теплой, а холодной, плюс к тому же она выглядела скорее как салат, в котором как оказалось участвовали яблоки и кажется даже виноград. Наверное любители здорового питания были в восторге. В дополнение была небольшая порция вполне обычного салата: кусочки болгарского перца, листочки салата, какая-то зелень (если честно, состав я уже не помню, надо было наверное все фотографировать. Век живи — век учись! :-) ). И был двойной бутерброд типа сэндвич, если я ничего не путаю то можно было выбрать либо с куриным мясом либо с копченым лососем. Я выбрал с лососем. В состав бутерброда также вошел какой-то творожный сыр, похожий на Филадельфию и опять же странным образом тонкие ломтики яблок. В добавок там был еще какой-то салатик и мини десерт. В общем, скромно и со вкусом. Мы однозначно утолили голод и даже получили какое-то эстетическое удовольствие.

    Если по поводу еды у меня в целом претензий не было, несмотря на немного необычный, на мой взгляд, подбор блюд. То по поводу кофе, а вернее его постоянного отсутствия у меня претензии были, есть и будут. Я не знаю как у вас с этим делом, но для меня кофе это топливо. Я не могу интенсивно работать без кофе. Любой проект, любая нагрузка — мне нужен кофе. И насколько я знаю таких как я в Канаде много. Кофе есть в каждом офисе где я работал, а я уже успел поработать примерно в десятке компаний, больших и малых, богатых и не очень. Кофе есть везде, где-то получше, где-то похуже, где-то нерегулярно, но кофе машина обязательный атрибут любого, уважающего себя офиса. Так вот, с поставками кофе на хакатон были большие проблемы. Кофе закончился уже к 12. С утра было 2 термоса наверное литра на 3 каждый. И я так и не понял обновляли ли запасы кофе после ланча или нет, но у меня реально была небольшая «ломка».

    After lunch, in anticipation of real data in the database, I focused on interacting with the front end. I asked Zvezdochka to show what they do with the integration with the Radio Canada page . She showed that all they managed to achieve at the moment was to clear completely original HTML, or the main block with content.





    And my dreams and fantasies about trying to reuse the original CSS are very unlikely. With this quality of design, I no longer saw the point of trying to integrate into the original page of Radio Canada. It turned out that with such clumsy integration, the page becomes uglier and prettier, as intended, but rather, on the contrary, we simply disfigure the original design with our inserts. While it is still unknown how the original JS and CSS will react.

    After a short discussion, everyone agreed to slightly change the concept of our prototype. I suggested dropping the ideb of integration, and since we have a browser plug-in anyway, we can output the results directly in it. That is, right below the search field, display a panel in the right part of the browser window.



    As time went. There was a hitch when Zvezdochka had to rewrite her cycles to make the number of categories equal to 2 and dynamically create sections for the category if a new one appeared on the next element and add content to the existing section if the category was already received earlier. Plus, the design for the right panel meant the output was no longer in 3 columns, but, just like a tape, in one.

    Plato was briefly stuck with data output to the database. As I noted earlier, this was for him the first example of the real use of python with a database. Hitch happened with prepared statement. When Plato was tired of fighting python's incomprehensible messages, he asked me to take a fresh look at what could be the problem. Here I was able to help, because the database is my topic, so to speak. Plato tried to modify an example from the Internet for a prepared statement. But he used direct concatenation of values ​​and naturally the python cursed. As soon as we separated the request from the data, everything worked.

    As soon as the first data appeared in the database, I replaced my data stubs with real data from the database. In this connection, the guys at the front end immediately floated styles and elements. The category headings, firstly, turned out to be too large, and secondly, it turned out that the data, with some fright, included randomly HTML markup.

    Data examples
    title:

    Protection de l’eau potable : des municipalités québécoises prêtes à aller devant les tribunaux

    summary:

    Des représentants de plus de 300 municipalités du Québec ont voté à l'unanimité une résolution qui leur permettrait d'aller devant les tribunaux si Québec n'impose pas des normes plus sévères afin de protéger leurs sources d'eau potable. Même s'ils concèdent voir des signes positifs au gouvernement provincial, ils menacent d'entamer des démarches judiciaires si les lois n'évoluent pas rapidement.


    body:

    Ma régionGrand Montréal

    Des représentants de plus de 300 municipalités du Québec ont voté à l'unanimité une résolution qui leur permettrait d'aller devant les tribunaux si Québec n'impose pas des normes plus sévères afin de protéger leurs sources d'eau potable. Même s'ils concèdent voir des signes positifs au gouvernement provincial, ils menacent d'entamer des démarches judiciaires si les lois n'évoluent pas rapidement.

    Avec ce vote unanime, les municipalités rassemblées ont envoyé un signe clair à Québec. Les 338 représentants demandent une dérogation, au ministre de l’Environnement, pour obtenir le pouvoir d'imposer des distances d'un minimum de deux kilomètrres entre les sources d'eau potable et les éventuelles installations gazières et pétrolières, plutôt que les 500 mètres prescrits par le gouvernement.

    « Ça m’a ému de voir l’unanimité et la solidarité de mes collègues », raconte Gérard Jean, maire de Lanoraie, une municipalité de Lanaudière.

    La mairesse Lisette Maillé, d'Austin, dans une salle de réunion

    C'est deux kilomètres ou rien!

    Lisette Maillé, mairesse d'Austin, en Estrie

    Actuellement, des pourparlers sont en cours avec la ministre du Développement durable, de l'Environnement et de la Lutte contre les changements climatiques, Isabelle Melançon.

    Les membres du comité de pilotage du dossier se disent confiants que la ministre Melançon approuve leur demande. À la suite d'une rencontre avec le comité, le 20 mars, elle s'est engagée à rendre une réponse dans la semaine du 9 avril.

    En entrevue avec Radio-Canada, Gérard Jean s’est montré optimiste, disant avoir senti « une grande ouverture » de la part de Mme Melançon.

    M. Jean rappelle que les municipalités attendent une réponse positive depuis près de quatre ans. Dans la négative, les municipalités pourraient envisager des mesures judiciaires.

    Un précédent

    Le 18 février, la municipalité de Ristigouche Sud-Est a gagné son procès face à la pétrolière Gastem. Le droit de protéger les sources d'eau potable de ses citoyens lui a été reconnu par la Cour supérieure. Selon le maire de Ristigouche Sud-Est, François Boulay, ce jugement permet d'« appuyer les municipalités » face à un règlement qu'il juge actuellement « inadéquat ».

    Comme nous l’avons vu dans le jugement de Ristigouche, le principe de précaution appartient aux municipalités. C’est nous qui sommes redevables aux citoyens.

    Gérard Jean, maire de Lanoraie

    Selon M. Jean, ce jugement « renforce » la position des municipalités dans ce dossier. « Je pense que le gouvernement a pris bonne note de ça », a-t-il ajouté.

    Vendredi, Mme Melançon s’est prononcée contre les projets d'exploitation de pétrole et de gaz naturel sur le territoire de la province.

    C'est deux kilomètres ou rien!

    Lisette Maillé, mairesse d'Austin, en Estrie

    Actuellement, des pourparlers sont en cours avec la ministre du Développement durable, de l'Environnement et de la Lutte contre les changements climatiques, Isabelle Melançon.

    Les membres du comité de pilotage du dossier se disent confiants que la ministre Melançon approuve leur demande. À la suite d'une rencontre avec le comité, le 20 mars, elle s'est engagée à rendre une réponse dans la semaine du 9 avril.

    En entrevue avec Radio-Canada, Gérard Jean s’est montré optimiste, disant avoir senti « une grande ouverture » de la part de Mme Melançon.

    M. Jean rappelle que les municipalités attendent une réponse positive depuis près de quatre ans. Dans la négative, les municipalités pourraient envisager des mesures judiciaires.

    Un précédent

    Le 18 février, la municipalité de Ristigouche Sud-Est a gagné son procès face à la pétrolière Gastem. Le droit de protéger les sources d'eau potable de ses citoyens lui a été reconnu par la Cour supérieure. Selon le maire de Ristigouche Sud-Est, François Boulay, ce jugement permet d'« appuyer les municipalités » face à un règlement qu'il juge actuellement « inadéquat ».

    Comme nous l’avons vu dans le jugement de Ristigouche, le principe de précaution appartient aux municipalités. C’est nous qui sommes redevables aux citoyens.

    Gérard Jean, maire de Lanoraie

    Selon M. Jean, ce jugement « renforce » la position des municipalités dans ce dossier. « Je pense que le gouvernement a pris bonne note de ça », a-t-il ajouté.

    Vendredi, Mme Melançon s’est prononcée contre les projets d'exploitation de pétrole et de gaz naturel sur le territoire de la province.

    En entrevue avec Radio-Canada, Gérard Jean s’est montré optimiste, disant avoir senti « une grande ouverture » de la part de Mme Melançon.

    M. Jean rappelle que les municipalités attendent une réponse positive depuis près de quatre ans. Dans la négative, les municipalités pourraient envisager des mesures judiciaires.

    Un précédent

    Le 18 février, la municipalité de Ristigouche Sud-Est a gagné son procès face à la pétrolière Gastem. Le droit de protéger les sources d'eau potable de ses citoyens lui a été reconnu par la Cour supérieure. Selon le maire de Ristigouche Sud-Est, François Boulay, ce jugement permet d'« appuyer les municipalités » face à un règlement qu'il juge actuellement « inadéquat ».

    Comme nous l’avons vu dans le jugement de Ristigouche, le principe de précaution appartient aux municipalités. C’est nous qui sommes redevables aux citoyens.

    Gérard Jean, maire de Lanoraie

    Selon M. Jean, ce jugement « renforce » la position des municipalités dans ce dossier. « Je pense que le gouvernement a pris bonne note de ça », a-t-il ajouté.

    Vendredi, Mme Melançon s’est prononcée contre les projets d'exploitation de pétrole et de gaz naturel sur le territoire de la province.

    M. Jean rappelle que les municipalités attendent une réponse positive depuis près de quatre ans. Dans la négative, les municipalités pourraient envisager des mesures judiciaires.

    Un précédent

    Le 18 février, la municipalité de Ristigouche Sud-Est a gagné son procès face à la pétrolière Gastem. Le droit de protéger les sources d'eau potable de ses citoyens lui a été reconnu par la Cour supérieure. Selon le maire de Ristigouche Sud-Est, François Boulay, ce jugement permet d'« appuyer les municipalités » face à un règlement qu'il juge actuellement « inadéquat ».

    Comme nous l’avons vu dans le jugement de Ristigouche, le principe de précaution appartient aux municipalités. C’est nous qui sommes redevables aux citoyens.

    Gérard Jean, maire de Lanoraie

    Selon M. Jean, ce jugement « renforce » la position des municipalités dans ce dossier. « Je pense que le gouvernement a pris bonne note de ça », a-t-il ajouté.

    Vendredi, Mme Melançon s’est prononcée contre les projets d'exploitation de pétrole et de gaz naturel sur le territoire de la province.

    Un précédent

    Le 18 février, la municipalité de Ristigouche Sud-Est a gagné son procès face à la pétrolière Gastem. Le droit de protéger les sources d'eau potable de ses citoyens lui a été reconnu par la Cour supérieure. Selon le maire de Ristigouche Sud-Est, François Boulay, ce jugement permet d'« appuyer les municipalités » face à un règlement qu'il juge actuellement « inadéquat ».

    Comme nous l’avons vu dans le jugement de Ristigouche, le principe de précaution appartient aux municipalités. C’est nous qui sommes redevables aux citoyens.

    Gérard Jean, maire de Lanoraie

    Selon M. Jean, ce jugement « renforce » la position des municipalités dans ce dossier. « Je pense que le gouvernement a pris bonne note de ça », a-t-il ajouté.

    Vendredi, Mme Melançon s’est prononcée contre les projets d'exploitation de pétrole et de gaz naturel sur le territoire de la province.

    Le 18 février, la municipalité de Ristigouche Sud-Est a gagné son procès face à la pétrolière Gastem. Le droit de protéger les sources d'eau potable de ses citoyens lui a été reconnu par la Cour supérieure. Selon le maire de Ristigouche Sud-Est, François Boulay, ce jugement permet d'« appuyer les municipalités » face à un règlement qu'il juge actuellement « inadéquat ».

    Comme nous l’avons vu dans le jugement de Ristigouche, le principe de précaution appartient aux municipalités. C’est nous qui sommes redevables aux citoyens.

    Gérard Jean, maire de Lanoraie

    Selon M. Jean, ce jugement « renforce » la position des municipalités dans ce dossier. « Je pense que le gouvernement a pris bonne note de ça », a-t-il ajouté.

    Vendredi, Mme Melançon s’est prononcée contre les projets d'exploitation de pétrole et de gaz naturel sur le territoire de la province.

    Comme nous l’avons vu dans le jugement de Ristigouche, le principe de précaution appartient aux municipalités. C’est nous qui sommes redevables aux citoyens.

    Gérard Jean, maire de Lanoraie

    Selon M. Jean, ce jugement « renforce » la position des municipalités dans ce dossier. « Je pense que le gouvernement a pris bonne note de ça », a-t-il ajouté.

    Vendredi, Mme Melançon s’est prononcée contre les projets d'exploitation de pétrole et de gaz naturel sur le territoire de la province.

    Vendredi, Mme Melançon s’est prononcée contre les projets d'exploitation de pétrole et de gaz naturel sur le territoire de la province.

    À lire aussi :

    La jeunesse américaine se mobilise en masse contre les armes à feu

    Attentat en France : hommage national en l’honneur du gendarme Beltrame

    Alexis Lafrenière : l’éclatante entrée en scène d’un surdoué

    Carles Puigdemont évite l'arrestation en Finlande

    Immigration : le silence de la CAQ

    Veuillez noter que Radio-Canada ne cautionne pas les opinions exprimées. Vos commentaires seront modérés, et publiés s'ils respectent la nétiquette. Bonne discussion!

    ....



    Upon discovering this, the Asterisk began to redo the output of content in itself in JS, and I naturally began to change at home. Since I thought that I am a content provider for the frontend, so even if I crawl dirty text from the database, I must clear it and return the text already cleared of HTML garbage.

    It’s good that in this case we did not need the body of the articles, as you can see the storage format very unexpectedly includes not only HTML markup and CSS styles inline, but also calls to JS functions. I would not dare call it an API, let alone a public API. But we have what we have for our prototype, it’s enough for us to overcome the title and summary, for this we quickly created a function that cleans up HTML garbage and subsequently produces the necessary formatting of other data.

    Format the data from the database before sending it to the frontend
    private function decorate($article) {
            return [
                'video_id'=>$article['article_id'],
                'category'=>$article['region'],
                'title'=> html_entity_decode($article['title']),
                'description'=>strip_tags(html_entity_decode($article['summary'])),
                'image'=>$article['images'],
                'rating'=>round(10*floatval ($article['sentiment']),0),
                'link'=>$article['link'],
                'tags'=>$this->decorateTags($article['keyphrases'])
            ];
        }


    Time is ticking. Everything is in operation and debugging. All parts of the prototype look functioning. The team feels an emotional lift.

    But, as was known from the very beginning, we need to somehow screw the AI ​​to our search line!

    So we have an “emotional assessment” and “key words” that the almighty AI returned to us. But how do we show the importance and necessity of these options in the search? Should we sort the search results by positivity rating? What to do with keywords? Moreover, because of the delightful "dirty" content, the keywords look, for example, like this:
    municipalités du Québec, signes positifs, gouvernement provincial, nbsp, sources d'eau potable, démarches judiciaires, lois, unanimité, représentants, normes, résolution, tribunaux

    Pay attention to the keyword "nbsp" ;-) Although, we must pay tribute to the AI, he did not seem to return HTML tags as keywords.

    On the go, it was decided that, regardless of how we use these additional analysis data, I definitely need to give them along with each record, as you already saw in the function a little higher.

    For visualization, I suggested using a pseudographic progress bar of some characters. Just because I didn’t want the guys to spend time searching for some plugins, especially since, as I already said, they worked with bare JS even without jQuery, so I doubted that they would be able to quickly find and fasten something.

    As for the keywords, they came up with turning them into hashtags links to Twitter. At first it seemed that Twitter would not have content for any arbitrary word or phrase, but, after several tests, it turned out that the idea was quite workable.

    Thus, at this stage, we decided that we would simply expand and enrich the search results with a rating of positivity and hashtag links.

    Somehow promptly and inconspicuously approached dinner time. Despite the fact that everyone was very immersed in the work, we all looked up from the code with pleasure and went to the room where the participants brought rations :-) I definitely don’t remember the menu anymore, but the dinner was cold again, which made me a little upset. And as I said with coffee, too, was stressful.

    During dinner, we discussed plans for ending the day, we still had a lot to debug. I suggested if we have the opportunity to add the ability to filter by level of positivity in our plugin. Like, if a user doesn’t want to read “bad” news today, then he can raise the minimum level of positivity when searching and we will give him filtered results. The guys agreed to try, because their functions were divided. The asterisk was mainly engaged in design and layout, and Mercury interacted with my PHP backend. And this part was, as it were, completed and worked, and we could quite manage to add one or two new requests.

    Plato discussed the need to process more data. He explained that 100-120 records that we have already received are just a category. And that he has the opportunity to pull out a few more categories and, accordingly, drive everything through the AI ​​and upload to the database. They decided on that.

    So after dinner, we all returned to work. But I asked Zvezdochka to buy coffee.
    Do not count for advertising, but rather a national flavor
    У нас в Канаде есть сеть кофейн TimHortons. Это канадская франшиза.



    Пока мы сюда не приехали я о них никогда не слышал. Но тут это практически один из символов Канады. В целом, в общественном мнении, все согласны, что самый качественный кофе в Starbucks, но Timhortons — это не только качественный кофе, но и поддержка своего отечественного канадского производителя. Плюс пончики (donuts) это тоже визитная карточка этой франшизы.

    Очень много народа начинает свой день именно в TimHortons. Люди заезжают туда по дороге на работу за кофе или кофе с пончиками или даже за полноценным завтраком, какой-нибудь омлет с беконом.



    Почему я решил рассказать о TimHortons? Потому что у них есть большая коробка кофе, я никак не могу запомнить название в меню. Но суть в том что вам выдается картонная коробка с горячим кофе на 12 человек, плюс стаканчики с крышечками, плюс сахар и сливки, плюс палочки для размешивания. Этот формат очень удобен для всяких посиделок на природе или просто в компании. Картонный термос вполне прилично держит температуру.



    And when Zvezdochka returned with a magical hot peak, we got a second wind.

    Over the next 3-4 hours, we worked hard trying to create something cute. For a dynamic search, I made two different requests, depending on the number of search characters that come to me on the backend in the request. Since I knew that full-text search does not work well with short words, and simply ignores many letter combinations, I decided that for a search of up to 3 characters I would use a regular search in the database LIKE, and when the search string becomes longer, I will execute another query with MATCH.

    This will allow us to avoid empty answers from the backend, for a more beautiful presentation.

    Do not judge strictly
    public function searchRange( $min, $max, $term) {
            $pdo = DB::connection()->getPdo();
            if(strlen($term)>3){
                $query = "SELECT * from articles WHERE 
                            ROUND(sentiment,1) >= :min AND ROUND(sentiment,1) <= :max
                            AND images != '0'
                            AND MATCH (title, summary, body, keyphrases) AGAINST (:term  IN NATURAL LANGUAGE MODE) LIMIT 50;";
                $stmt = $pdo->prepare($query);
                $stmt->bindValue('min',round(intval($min)/10,2));
                $stmt->bindValue('max',round(intval($max)/10,2));
                $stmt->bindParam('term',$term);
            } else {
                $query = "SELECT * from articles WHERE 
                            ROUND(sentiment,1) >= :min AND ROUND(sentiment,1) <= :max
                            AND images != '0'
                             AND (title LIKE :term
                             OR summary LIKE :term1)
                          LIMIT 50;";
                $stmt = $pdo->prepare($query);
                $stmt->bindValue('min',round(intval($min)/10,2));
                $stmt->bindValue('max',round(intval($max)/10,2));
                $stmt->bindValue('term',"%$term%");
                $stmt->bindValue('term1',"%$term%");
            }
            $stmt->execute();
            $res = $stmt->fetchAll();
            $ret = [];
            foreach($res as $article) {
                $ret[] = $this->decorate($article);
            }
            return response()->json($ret,200)->header('Access-Control-Allow-Origin',"*");
    }


    But we all already experienced joy and, probably, even pride in the fact that the solution worked. We had a full-fledged prototype that worked according to our plan. It met all the criteria announced by the organizers. This was more than enough for all of us.

    At the very end of the day, Plato provided approximately 1,200 entries in a table of 80 categories / regions. For a quick test and demonstration, the result looked at least working.



    Concluding the next part of my story, I want to apologize, firstly, for the delay (almost a month has passed since the publication of the second part), believe me, I sincerely tried to write faster. During this time, I managed to lose my job, find a new one and learn, though fluently, ELK stack (Elasticsearch, Logstash, Kibana, Filebeat). I think the next article will be about Elastic.

    When I started writing this 3rd part, I was sure that it would be final. I overestimated my strength and underestimated the amount of material. Probably with experience I will learn to express something more succinctly. Please take into account that in this case it is difficult to determine what exactly will interest the reader. This is still not quite a technical article, but a little about everything. But at the same time I want to show the technical side of what is happening.

    I almost forgot about, you can download and install our Chrome plugintrue only in developer mode. You can view the code before installation, thank God, there is practically nothing there and read something.

    I will try very hard not to linger on the final part with a description of the final day, the finalists' projects and the general conclusions that I made for myself.

    Only registered users can participate in the survey. Please come in.

    Which article of the author would you like to read next

    • 71.4% Hackathon - Final 10
    • 28.5% How to raise and configure Elasticsearch, Kibana, Logstash, Filebeat (first acquaintance) 4

    Also popular now: