Tale of how the AR SDK was searched on the hackathon
In the kingdom of the thirties in the thirtieth state ... I began my story about the hackathon in Wrike because Hackathon is like a fairy tale: enthusiasts gather to give life to their ideas. An idea is being released, like a fairy-tale arrow, and then it may fall on the boyar's courtyard, or perhaps disappear in the swamp of everyday life. And, as in a fairy tale, it is always exciting. It’s not just a very short time to assemble a team and make a product out of an idea, and also show it so that honest people surprise.
But seriously, I want to share the experience of participation in the hackathon, where we have developed our augmented reality (AR). I'll tell you about how we tried to find a ready-made AR SDK for our task, but we could not. In the end, we decided to write AR ourselves and succeeded.
I love hackathons very much: I took part in several, I had to do it myself, and if I could, I would take part.
Hakatons are often conducted by specific organizations, and this works for the company's HR brand. The goals may be different: a story about a company or a product, hiring different degrees of aggressiveness, organizing a thematic community, searching for fresh ideas ( despite the armies of their own productologists, it may be useful to get a bunch of ideas from dreamers with a straightforward look at the subject area ) and so on. d.
For participants it is an opportunity to get acquainted with the company, because often a hackathon is its reflection, and it is possible to draw conclusions about the internal kitchen. To understand the processes in a company, you can look at how it organizes a hackathon: what are the limitations for projects participating in a hackathon ( scope of tasks, subject area, technology, tools, etc. ); the level of the event; criteria and transparency of the competitive part; refereeing - composition and quality; what are the rules and methods of forming teams. In general, companies are divided into those who conduct hackathons ( internal or public ) and those who do not. I prefer those who conduct, because these are more open companies.
I myself go to hackathons not for the sake of victory, but for the sake of participation. I am interested in:
Therefore, I am happy to participate in the wrike hackathons ( this year was already the third internal hackathon ), where we invent and make wrike even better: some of the previous hackathon projects already live in our product, and some are in team backlogs. Inspire and scale, despite the fact that the hackathon is internal, about 30 teams are recruited ( more than 100 people ) - all with fresh cool ideas.
On hackathon 2018, I decided to try working with AR . In MVP, I wanted to display wrike tasks ( name, status, artists, etc.) on the mobile phone screen when you hover on a graphic code with the task ID encrypted in it, and also add the ability to change the status and assign / remove the task from yourself. There is an idea, there is a hackathon, the case didn’t follow the team either. On the appointed day everything began to turn.
I don’t particularly prepare for hackathons in terms of setting up the environment ( searching for SDK and frameworks; installing software; configuring, etc. ), writing code in advance, etc., but only working on ideas, features, thinking what to do. etc. Therefore, we conferred with the team that we would write in Java ( it’s written on Android natively on it ), and it was hypothesized that it was probably full of ready-made AR libraries. Our plan is to take a convenient SDK, attach the Wrike API to it , and then focus on writing the logic of our application. Thus, our first task was to find a convenient Java AR SDK, which allows you to:
Looks like a pretty simple task. And we started sorting through options based on articles like " Top Augmented Reality SDK in 2018 "
First, we turned our views on Google. They opened the " Quick Start ", did everything according to the instructions, launched and, oh, a miracle, everything works: I have Androids on my desk, which can also be moved. The feeling that we have found the "foundation" for our application. But further disappointment followed, image recognition does not work the way we need: there can be only one image, it should be clearly visible, and it should be from the database of previously known images ( and we should have our own unique marker for each task ). And the saddest thing is the inability to control the focus, which makes it difficult for the user to catch the image we need for recognition. The truth is now with the problem solved, and we had to continue the search. In general, Google was good, but not for our task. And because of the specifics of the work of OpenGL on OSX, we could not make the demo work in the emulator and did everything on a live phone.
Read the documentation, watched the videos , it looks impressive. Many features, for example, Image Targets . We decided to try: registered, downloaded, collected, launched. Demo application started, but neither on the emulator, nor on live android-e did not work. Attempting to verify any function caused the entire application to crash. It was decided not to waste time searching for a problem and fixing it, and moving on to the next SDK.
We downloaded the SDK, went to the tutorial , launched the demo. There are a lot of opportunities here, the demo is immediately impressive, a lot of mini examples, we have played enough ( for example, there is face recognition ), and, oh, a miracle, the demo already has QR recognition. But the problem is that we get what is encrypted in the code, but do not know where it is located. Began to understand how the QR scanner is arranged. It turned out that it was made as a superstructure over ZBar in the form of a plus plug-in to the SDK. First, a crazy thought came to uncover the gcc and finish the plug-in, so that it also gave the coordinates, but we stopped in time.
Realizing that a good quarter of the time of the allotted days is behind, and we are still looking for our SDK (there were tests and other solutions, not only described above, but also there was a fiasco ), we decided not to look for a “silver bullet” anymore, but to take everything your hands A new plan has matured: as a task marker, we take a QR code, as simple and common; for their recognition, we take ZXing , which can recognize several codes at the same time and besides the value, the library also gives the coordinates of 3 “search” points of the QR code. And then on top of the code reader we will implement our AR. We rolled up our sleeves and forward, we have 3 points, which means that with the help of affine transformations we can get everything we need.
We did not begin to search for a library for mathematics, since the task is not difficult. First of all, we made our class for the necessary recalculation of coordinates. The final algorithm for working with QR code turned out to be rather primitive:
We added an algorithm, it works, we test, we struggle with memory leaks, we add visual effects, we get pleasure from the hackathon.
In hackathons, besides your product itself, it’s very important how you present it. Everyone understands that you have a very short time and a technically beautiful solution is not expected of you, so you need to show the beauty of your idea. I always like the story-telling approach, where it is clear to listeners, for whom the product is made, under what conditions it is used and which task it solves.
Therefore, in our presentation, in addition to demonstrating the obtained functionality, we called on the power of imagination and described situations in which such augmented reality ( AR glasses will soon become commonplace) can improve the lives of those who work not with a computer, but whose work is connected with wrike. For example, for the interaction between the designer-architect engaged in repairing the house, and the team that directly implements the project in the house itself.
I think we honorably demonstrated our MVP, and caught our rays of love. Summer is over and picnics and holidays are coming to an end, and we are thinking of devoting autumn evenings to the development of our Wrike AR.
Thanks for the illustrations Sai Kin !
But seriously, I want to share the experience of participation in the hackathon, where we have developed our augmented reality (AR). I'll tell you about how we tried to find a ready-made AR SDK for our task, but we could not. In the end, we decided to write AR ourselves and succeeded.
A tip
I love hackathons very much: I took part in several, I had to do it myself, and if I could, I would take part.
Hakatons are often conducted by specific organizations, and this works for the company's HR brand. The goals may be different: a story about a company or a product, hiring different degrees of aggressiveness, organizing a thematic community, searching for fresh ideas ( despite the armies of their own productologists, it may be useful to get a bunch of ideas from dreamers with a straightforward look at the subject area ) and so on. d.
For participants it is an opportunity to get acquainted with the company, because often a hackathon is its reflection, and it is possible to draw conclusions about the internal kitchen. To understand the processes in a company, you can look at how it organizes a hackathon: what are the limitations for projects participating in a hackathon ( scope of tasks, subject area, technology, tools, etc. ); the level of the event; criteria and transparency of the competitive part; refereeing - composition and quality; what are the rules and methods of forming teams. In general, companies are divided into those who conduct hackathons ( internal or public ) and those who do not. I prefer those who conduct, because these are more open companies.
I myself go to hackathons not for the sake of victory, but for the sake of participation. I am interested in:
- Try new technologies. On one “food” hackathon we took Flutter , and wrote the application under ios and android. Although none of us have tried Flutter before, the truth is thatwe were able to Dart .
- To get to know and work with new people , so after one of the “urban” hackathons, I called an accomplice of the hakaton project to work in my team for the main job, which I never regretted. Hackathon is a great way to check out a comrade in action.
- To do something that I really need myself. On the internal hackathon they wrote down the application, which was later used in the work.
- Just get positive emotions from creation. I really like the hackathon atmosphere!
Therefore, I am happy to participate in the wrike hackathons ( this year was already the third internal hackathon ), where we invent and make wrike even better: some of the previous hackathon projects already live in our product, and some are in team backlogs. Inspire and scale, despite the fact that the hackathon is internal, about 30 teams are recruited ( more than 100 people ) - all with fresh cool ideas.
On hackathon 2018, I decided to try working with AR . In MVP, I wanted to display wrike tasks ( name, status, artists, etc.) on the mobile phone screen when you hover on a graphic code with the task ID encrypted in it, and also add the ability to change the status and assign / remove the task from yourself. There is an idea, there is a hackathon, the case didn’t follow the team either. On the appointed day everything began to turn.
I asked ash
I don’t particularly prepare for hackathons in terms of setting up the environment ( searching for SDK and frameworks; installing software; configuring, etc. ), writing code in advance, etc., but only working on ideas, features, thinking what to do. etc. Therefore, we conferred with the team that we would write in Java ( it’s written on Android natively on it ), and it was hypothesized that it was probably full of ready-made AR libraries. Our plan is to take a convenient SDK, attach the Wrike API to it , and then focus on writing the logic of our application. Thus, our first task was to find a convenient Java AR SDK, which allows you to:
- Draw something on a given virtual surface.
- Integrate / already contains a scanner of dynamic graphic elements ( barcode, QR, etc. ).
- Work with a low entry threshold ( we are on the hackathon, we need to quickly ): there is a demo, there is documentation, there is a free / trial version.
Looks like a pretty simple task. And we started sorting through options based on articles like " Top Augmented Reality SDK in 2018 "
Google's ARCore
First, we turned our views on Google. They opened the " Quick Start ", did everything according to the instructions, launched and, oh, a miracle, everything works: I have Androids on my desk, which can also be moved. The feeling that we have found the "foundation" for our application. But further disappointment followed, image recognition does not work the way we need: there can be only one image, it should be clearly visible, and it should be from the database of previously known images ( and we should have our own unique marker for each task ). And the saddest thing is the inability to control the focus, which makes it difficult for the user to catch the image we need for recognition. The truth is now with the problem solved, and we had to continue the search. In general, Google was good, but not for our task. And because of the specifics of the work of OpenGL on OSX, we could not make the demo work in the emulator and did everything on a live phone.
Vuforia
Read the documentation, watched the videos , it looks impressive. Many features, for example, Image Targets . We decided to try: registered, downloaded, collected, launched. Demo application started, but neither on the emulator, nor on live android-e did not work. Attempting to verify any function caused the entire application to crash. It was decided not to waste time searching for a problem and fixing it, and moving on to the next SDK.
Wikitude
We downloaded the SDK, went to the tutorial , launched the demo. There are a lot of opportunities here, the demo is immediately impressive, a lot of mini examples, we have played enough ( for example, there is face recognition ), and, oh, a miracle, the demo already has QR recognition. But the problem is that we get what is encrypted in the code, but do not know where it is located. Began to understand how the QR scanner is arranged. It turned out that it was made as a superstructure over ZBar in the form of a plus plug-in to the SDK. First, a crazy thought came to uncover the gcc and finish the plug-in, so that it also gave the coordinates, but we stopped in time.
And they fought 3 days and 3 nights
Realizing that a good quarter of the time of the allotted days is behind, and we are still looking for our SDK (there were tests and other solutions, not only described above, but also there was a fiasco ), we decided not to look for a “silver bullet” anymore, but to take everything your hands A new plan has matured: as a task marker, we take a QR code, as simple and common; for their recognition, we take ZXing , which can recognize several codes at the same time and besides the value, the library also gives the coordinates of 3 “search” points of the QR code. And then on top of the code reader we will implement our AR. We rolled up our sleeves and forward, we have 3 points, which means that with the help of affine transformations we can get everything we need.
We did not begin to search for a library for mathematics, since the task is not difficult. First of all, we made our class for the necessary recalculation of coordinates. The final algorithm for working with QR code turned out to be rather primitive:
- The image from the camera is transferred to ZXing, we get an array with the coordinates of the points and the values of the QR code.
- From the 3 coordinates, we calculate the 4th angle of the square, increase the square one and a half times to cover the initial QR code, and get the basis for the card.
- Make a request to the Wrike API to retrieve data about the task.
- We draw a card, thanks to affine transformations, we save all distortions ( viewing angle, rotation, scale ).
We added an algorithm, it works, we test, we struggle with memory leaks, we add visual effects, we get pleasure from the hackathon.
And I was there, honey-beer drank
In hackathons, besides your product itself, it’s very important how you present it. Everyone understands that you have a very short time and a technically beautiful solution is not expected of you, so you need to show the beauty of your idea. I always like the story-telling approach, where it is clear to listeners, for whom the product is made, under what conditions it is used and which task it solves.
Therefore, in our presentation, in addition to demonstrating the obtained functionality, we called on the power of imagination and described situations in which such augmented reality ( AR glasses will soon become commonplace) can improve the lives of those who work not with a computer, but whose work is connected with wrike. For example, for the interaction between the designer-architect engaged in repairing the house, and the team that directly implements the project in the house itself.
I think we honorably demonstrated our MVP, and caught our rays of love. Summer is over and picnics and holidays are coming to an end, and we are thinking of devoting autumn evenings to the development of our Wrike AR.
Thanks for the illustrations Sai Kin !