Navigation in the store: through augmented reality to the desired shelf

We recently launched a pilot project: in our 10 Moscow stores you can quickly navigate the trading floors with the help of a small augmented reality application M.Go and immediately find shelves with the necessary groups of goods. As far as we know, this is the first such project in the world retail practice. I want to tell you more about this application, how it works and what this project can grow into if customers like it.

About 8-10 years ago, various augmented reality projects first appeared on smartphones. They made it possible to find on the image, static or video, a specific flat marker, similar to a QR code. The application could recognize its size and position in space. Since then, this technology has been actively developed. The loudest example of an augmented reality product is the Pokemon GO game, which blew up all the charts a few years ago. Also on the market today there are solutions that greatly facilitate third-party developers creating AR products.

How M.Go works

When launched, the application automatically determines your location and offers to choose the store in which you are located. After that M.Go will ask you to calibrate on the marker. They are glued on the floor in all 10 stores that today support the application (two markers in each, except for a large two-story store on Sadovaya-Spasskaya - you will already find three markers in it):


You need to point the smartphone’s camera at this sticker on the floor, the application will recognize it and “understand” exactly where you are standing.


And after a couple of seconds, a list of categories of goods and services available in this store will appear on the screen.


Click on the desired icon and two pointers will appear on the screen. One “hangs” in the distance above the place where you need to get (it can disappear from the screen if the camera is pointing to the other side), and the second pointer is always at the bottom of the screen - this is an arrow showing the direction like a navigator.






Why are several markers pasted in each store? The fact is that the application works correctly while it is active. As soon as you minimize it or another application becomes active (for example, someone calls you), navigation inside the store will be interrupted. And to use M.Go again, you need to calibrate. And so that customers do not have to return to the beginning each time, we pasted on an additional marker in stores.

How M.Go works

When we started creating the application, we relied on our vision and common sense. After all, there were no other similar decisions in the retail sector at that time. There were only good examples of using the technology stack to create augmented reality applications. We spent several private pilots. They tested the operation of AR technology and various interface options, which is especially important for such applications.

The first option was with an arrow showing the direction. In the second version there was a flying bot that moved along a predefined route. We chose the option with an arrow, so you can come to the desired department of the store, select a different category of goods and move on. And in the version with the bot, it was necessary to return again to the calibration marker, select the next category of goods and only after that go along a new route. Inconveniently. In addition, our stores are already arranged quite simply, this is not a maze of grocery hypermarkets in which you get tired of getting to the exit.

Today on the market there are two libraries ARKit and ARCore, created in the bowels of Apple and Google. Libraries can recognize not only markers, but also the space in which the user's device is located. Each frame from the camera is switched to black and white with high contrast, so that only contour lines remain on the image. The system finds the key points (the intersection of the contours in the image) and turns them into a "point cloud". These are spatial markers.

This technology formed the basis of our application.

On top of the ARCore and ARKit engines, we made a special architecture for working with storerooms. They have their own specifics - the rows of products are quite homogeneous and not always visually unique. The longer the customer walks around the store, the more the positioning error accumulates. We correct it in many ways. One solution is additional markers that allow you to recalibrate without returning to the beginning.

By the way, we first made calibration markers red-white, in company colors. They were very beautiful, but very quickly got dirty. I had to choose a different color combination, after which the markers even became more reliably recognized by the application.

In terms of CPU load, M.Go is comparable to some messenger. The main consumer of energy is the camera. M.Go can work on most iOS and Android devices, starting with Android 7.0 and iOS 11. This restriction is imposed on the SDK by Google and Apple themselves.

Why does our AR application “know” only 10 stores? Implementing support for each store requires a lot of work:

• make a digital planogram (store map) indicating the placement of all categories of goods,
• agree on where the markers will be placed,
• Test how accurately the application "navigates".
  

But the project is still a pilot, we want to understand whether people need such functionality.

What's next?

If the pilot is successful, we will embed AR functionality in our main application and add new features. There are many options. For example, to simplify the change in the placement of goods in the store. So that the administrator on the tablet simply indicates on the store diagram where these or those categories now lie. And this data would be synchronized with the application installed by users.

You can create a complete universal system for store managers, so that it:

• divide goods into subcategories in more detail,
• indicate the balance of goods
• add topical promotions.

We plan to add a seller’s call button to a specific area in the client application, as well as introduce gamification elements: catch discounts flying around the room or look for virtual bonuses behind the refrigerator. In addition, the solution will allow you to find goods that are not exhibited on the trading floor, with a detailed description and a 3D model.