Magic Leap released the SDK. Let's see what's inside?
Magic Leap is a startup dedicated to augmented reality devices. The company has already attracted $ 1.4 billion in investments from impressive investors, including Qualcomm, Google and the Alibaba Group. In 2016, Forbes valued Magic Leap at $ 4.5 billion. At that time, only renderings were shown to Magic Leap to the public.
Magic leap one
The ML1 kit includes the glasses themselves, which the manufacturer calls Lightwear, a Lightpack computing computer and a controller.
The manufacturer promises that the device will be light enough and convenient for several hours of continuous use. Digital Lightfield technology implemented in this device allows you to add augmented reality objects taking into account the environment.
Lumin os
As the operating system in Magic Leap One, Lumin OS is used. It is based on the Linux kernel. Part of the functionality was taken from Android, some were written by ourselves, since it was necessary to provide high performance for spatial computing tasks. The OS supports Wi-Fi, Bluetooth, multimedia and multitasking. Before using a microphone, camera and other devices related to personal information, the user will see a dialog box asking for access to them.
As 3D-engines, you can use Unreal Engine 4 or Unity, which are now standards in this area. You can also use the Lumin SDK C API to integrate the Lumin OS features into your engine.
Lumin sdk
The Lumin SDK includes a library that provides access to the capabilities of the ML1 device, a device simulator and a space simulator that allows you to generate a room in which the device simulator will be launched.
Platform features
Headpose
Allows you to get the coordinates and direction of ML1 in the real world. For tracking, VIO (cameras + accelerometer) is used. If there are problems with the cameras (low light, poor texture, overlapping), the movement tracking is disabled and only direction tracking works.
An interesting feature is that ML1 remembers the space around it. This allows you to restore the coordinates after they are lost and moved to another part of the room. You can also save the coordinates of the objects of augmented reality in a specific place in the room and restore them with a new start.
Gestures
ML1 can recognize 8 gestures. As information, you will be given the coordinates and probability, while the hands should not overlap each other. In the simulator, you will receive information about gestures that fall into the camera visibility range but are not visible to the user. On a real device, you will receive information only about gestures that are in the user's visibility zone.
Audio
SDK allows you to play "spatial" sound, taking into account the locations around the user. Without processing, sounds will appear flat, not enough to feel immersed. Soundfield Audio technology can be connected to projects on Unity, Unreal Engine 4 or through a plug-in to Wwise or FMOD. In future releases, it will be possible to use calls to the Lumin SDK directly.
Multimedia
ML1 supports playing audio and video using a media player. HTTP Live Streaming and RTSP are also supported. MPEG-DASH and Smooth Streaming are not yet supported.
World reconstruction
ML1 scans the space around it in real time. Data is collected from several positions, and then combined to reject noise. Three types of data are available to developers:
- getting tops;
- getting planes;
- getting the point of intersection of the beam with the object (raycast).
By default, recognized real-world objects will overlap augmented reality objects.
Recommendations for use:
- ML1 expects a static environment. If there is too much movement around you, there may be surprises;
- for a better scan of the space, walk around the room;
- do not move too fast;
- recommended distance to sensors from 40 cm to 5 meters;
- sunlight and other infrared light can adversely affect the operation of cameras;
- sensors may not recognize black, shiny, and light-absorbing objects;
Simulator
The SDK also includes Virtual Room Generator and Magic Leap Remote. Magic Leap Remote - simulator device ML1. Virtual Room Generator - a generator of rooms that can be transferred to the simulator.
The simulator allows you to work with audio, move the device in a virtual room, work with a virtual trackpad, recognize gestures, track images and receive data about room objects using raycast.
The company promises to deliver its devices to consumers this year. You can still have time to prepare your project and be on the list of the first applications in the store by the time sales start.
We will wait for news from Magic Leap and the answers of competitors. Currently, only Microsoft's HoloLens is among them. But a few days ago, Apple opened registration for the WWDC conference, whose logo this time is very similar to 3D interfaces:
