Doom of SceneKit. Yandex experience with 3D graphics in iOS

- I'm too young to die.

SceneKit is a high-level three-dimensional graphics framework in iOS that helps create animated scenes and effects. It includes a physics engine, a particle generator and a set of simple actions for 3D objects that allow you to describe the scene in terms of content — geometry, materials, lighting, cameras — and animate it through a description of changes for these objects.

Today we will look at SceneKit with an attentive, slightly harsh look, but first we turn to the basics and see what the 3D scene is and what needs to be done to create it.

The simplest scene of three nodes with geometry in them

First you need to create the basic structure of the scene, which consists of nodes or nodes of the scene. Each node can contain both geometry and other nodes. Geometry can be as simple as a ball, a cube, or a pyramid, or more complex, created in external editors.

Overlay materials

Then for this geometry it is necessary to specify materials that will determine the basic representation of the objects. Each material sets its own lighting model and, depending on it, uses a different set of properties . Each such property is usually a color or texture, but in addition to these commonly used options, there is also the possibility to use CALayer , AVPlayer , and SKScene .

Add light sources

After that, you need to add light sources that determine how well objects are visible in one or another part of the scene. They, by analogy with geometry, must lie inside some kind of node. SceneKit supports many different types of lighting , as well as several types of shadows .

Bokeh effect "out of the box"

Then you need to create a camera (and put it in a separate node) and set basic parameters for it. There are a lot of them, but with the help of them you can create cool effects. Out of the box, bokeh (or blur), HDR with adaptation, glow, SSAO and hue / saturation modifications are supported.

Simple animations in SceneKit

Finally, SceneKit includes a simple set of actions for 3D objects that allow you to set scene changes over time. SceneKit also supports actions described in JavaScript , but this is a topic for a separate article.

The interaction of the particle generator with the physics engine can lead to a tornado!

In addition to graphics, the main features of SceneKit are the particle generator and the advanced physics engine, which allows you to set real physical properties to both ordinary objects and particles from the generator.

About all these chips written a large number of detailed tutorials. But in the development process, we practically did not use these opportunities ...

Hey, not too rough

Once I wrote a lighting model for 3D games better than real sunlight, giving an acceptable FPS on the Nvidia 8800, but I decided not to release the engine, since God is cute to me and I don’t want to show its incompetence in this matter.
- John Carmack

We will begin a detailed study with a fairly simple task, which arises in almost everyone who works quite seriously with SceneKit: how to load a model with a complex geometry and connected materials, lighting, and even animations?

There are several ways, and they all have their pros and cons:

SCNScene (named :) - gets the scene from the bundle,
SCNScene (url: options :) - loads the scene by URL,
SCNScene (mdlAsset :) - converts a scene from different formats,
SCNReferenceNode (url :) - lazily loads the scene.

We get the scene from the bundle

You can use the standard method : put our model in dae or scn format in the scnassets bundle and load it from there by analogy with UIImage (named :).

But what if you want to control the update of the models yourself, without releasing an update in the App Store every time you need to change a couple of textures? Or imagine that you need to support user-created maps and models. Or - that you simply do not want to increase the size of the application, since the 3D-graphics in it is not the main functionality.

Load the scene by URL

You can use the scene constructor from the scn-file URL . This method supports downloading not only from the file system, but also from the network, but in the latter case, you can forget about compression. Plus, you will need to convert the model to scn format in advance. You can, of course, use dae, but with it comes a set of restrictions. For example - the lack of physically based rendering.

The main advantage of this method is that it allows you to flexibly configure import parameters . You can, for example, modify the life cycle of animations and make them endlessly repeated. You can explicitly specify the source for loading external resources such as textures, you can convert the orientation and scale of the scene, create missing normals for the geometry, merge all the geometry of the scene into one large node, or discard all non-standard elements of the scene.

Convert a scene from different formats.

The third option is to use a constructor with MDLAsset . That is, we first create the MDLAsset that is available in the ModelIO framework and then pass it to the designer for the scene.

This option is good because it allows you to download many different formats. Officially, MDLAsset can load obj, ply, stl, and usd formats, but after running out a list of all possible formats related to computer graphics, I found four more: abc, bsp, vox and md3, but they may not be fully supported or not. in all systems, and for them you need to check the correctness of the import.

It is also necessary to take into account that this method has an overhead for conversion, and use it very carefully.

These methods have one common pitfall: they return SCNScene, not SCNNode. The only way to add content to an already existing scene is to copy all the child nodes and — you can easily skip this step — animations from the root node (for example, they may appear when working with dae). In addition, you need to take into account that there can be only one environment texture in the scene (if you do not use custom shaders for reflections).

Lazily loading the scene

The fourth option is to use SCNReferenceNode . It returns not the scene, but the node, which can lazily (or on demand) load the entire scene hierarchy into itself. Thus, this method is similar to the first, but it hides within itself all the problems with copying.

He has one thing: the global parameters of the scene are lost.

It turns out that this is the easiest and fastest way to load your model, but if you need file-tuning, the first method will be better.

As a result, we stopped at the first option, since it was most convenient for us to work in the scn format, and for the designers to convert it from the dae format. In addition, we needed fine-tuning animations when loading.

Not premature optimizations at all

Having fiddled with this process long enough, I can give you some tips.

The main advice is to convert the files to scn beforehand. Then you can, by opening the file in Xcode's built-in scene editor, see exactly how your object will look in SceneKit.

In addition, the scn-file is actually just a binary representation of the scene, so downloading from it will take the least time. For the same dae, you first need to parse the xml, then convert all the meshes, animations and materials. Moreover, the conversion of animations and materials is a potential source of problems. We recall the lack of PBR support in dae: it turns out that if you want to use it, you have to change the type of all materials after the conversion and manually set the appropriate textures.

With this operation, you can get a very useful side effect: significant compression of textures. It is enough to open them in the "View" and export, changing the format to heic. On average, this simple operation saved 5 megabytes per model.

Also, if you download a scene from the Internet, I can advise you to download it in the archive, unpack it and pass the URL of the unpacked scn-file. This will save you and the user extra megabytes - which, in turn, will speed up the download, as well as reduce the number of points of failure. Agree: to make a separate request for each external resource, and even on the mobile Internet - not the best way to increase reliability.

Hurt me plenty

When I travel by car, I often hear the hard drive of the universe crackling, loading the next street.
- John Carmack

So, when the work on loading and importing models is put on stream, a new task arises: adding various effects and possibilities to the scene. And believe me, there is something to tell about. We'll start by going through the various collections in SceneKit.

The counters in SceneKit are considered immediately after physics. And before rendering the frame

Constraints, you say? What are the constraints? Few people know, and even more so, and talks about it, but SceneKit has its own set of constraints. And although they are not as flexible as the UIkit frameworks, with them you can still do a lot of interesting things.

SCNReplicatorConstraint

Let's start with a simple framework - SCNReplicatorConstraint . All he does is duplicate the position, rotation and size of another object with additional offset sets. Like all other constraints, it is possible to change the force and set the increment flag. Best of all, both parameters can be shown on this count.

Reduced strength 10 times

Strength influences how much a transformation is applied to an object. And once the position of the target object changes every frame, the shadow object approaches one-tenth of the difference in distance. Because of this, a lag effect appears.

Removed incrementality and reduced force 10 times.

Incrementality , in turn, affects whether the constraint is canceled after rendering. Suppose we turned it off. Then we see that the frame is applied on each frame before rendering, and after rendering it is canceled, and each frame is repeated this way. As a result, combining these two parameters, you can get a rather interesting effect of a clock hand.

The plane always faces the camera.

Let us turn to a more interesting construction: the so-called billboard.

Suppose it is necessary that some object is always to us "face". To do this, you just need to use SCNBillboardConstraint , specify which axis the object can rotate around. Further, before rendering each frame (after a step with physics), the positions and orientations of all objects will be updated to suit all constraints.

Here you can mention Look At Constraint : it is similar to a billboard, only the object can be placed facing any other object of the scene instead of the current camera.

What can be done with them? Of course, most often these frameworks are used to draw trees or small objects. Also due to them create special effects like fire or explosion. In addition, with their help, you can force the camera to follow the object on the stage.

Keeps distance between objects

SCNDistanceConstraint allows you to set the minimum and / or maximum distance to the position of another object. And yes, with it you can make a snake. :) This constraint can also be used to bind a camera to a character, although the camera position is usually more difficult to set, and it is not an easy task to describe it with some constraints. The same effect can be achieved by adding a spring in the physics engine, but this spring can be supplemented with a constraint in case you need to avoid problems with excessive stretching or compression of the spring.

Many have seen it in some Hitman, Fallout, or Skyrim: you drag a body behind you, it touches an obstacle - and it starts behaving as if a demon had infiltrated it. This would help to avoid such bugs.

SCNSliderConstraint

SCNSliderConstraint allows you to set the minimum distance between a given object and physical bodies with a suitable collision mask. Quite a funny constraint, but again, they try to simulate it through physical interaction. The basic idea is to set the radius of the dead zone with physical bodies for an object that does not have a physical body.

Inverse kinematics at work

SCNIKConstraint is the most interesting, but also the most complex, using the so-called inverse kinematics. Using a chain of parental nodes, inverse kinematics iteratively tries to bring the node to which you apply this constraint to the required point. In fact, it allows you not to think about the position in which the shoulder and forearm should be located, but simply to set the position of the hand and the possible angles of rotation of the connecting nodes. The rest will be counted for you. The main drawback of this constraint is that it allows you to set only the position of the hand, but not its orientation, and restrictions on the angles can be made global, without breaking down the axes.

So, we got acquainted in detail with the foundations and what they can do. Let's continue to explore interesting effects. We will deal with the effect of shadows.

Here is the plane, but it is not

It would seem, what could be simpler in the engine that supports shadows, than creating shadows? But sometimes the shadows need to be dropped on a completely transparent plane. This is very useful in ARKit, since the image of the camera is displayed behind the plane, and the shadow should be cast somewhere. The trick turns out to be quite simple: you must first turn on the pending shadows and turn off recording to all components near the plane in the material tab, and the shadow will continue to be superimposed on it. The only problem is that this plane will overlap the objects behind it.

But shadows are not the only poorly studied effect in SceneKit. Let's now deal with mirrors.

SCNFloor mirror - what could be easier

Everyone who has been playing with SceneKit probably knows about scnfloor, which adds mirror reflections to the floor. But for some reason, very few people use it for honest mirror reflections, because you can put your modelka over the floor geometry, tilt it a bit and turn it ... into an ordinary mirror.

Glass drips and curved mirror

But, what is even less known, you can set a normal map for this gender. Due to this, in turn, you can create many different interesting effects, like the effect of streaks or a curved mirror.

Ultra-violence

Once I was kissing a girl with open eyes. The middle clipping plane cut her face. Since then, I kiss only with my eyes closed.
- John Carmack

Shadows, mirrors - interesting effects. But there is one effect that with skillful use may be even more interesting - video textures.

Normal video and video with height map

You may need them just to show the video inside the game. But it is much more interesting that with the help of video textures you can modify the geometry. To do this, you need to put a video texture with a height map in the displacement property of your material and use the material on a plane with a sufficiently large number of segments . It remains to understand how to put it there.

I mentioned in the description of the scene creation process that you can use SKScene as the material property , and this is the SpriteKit scene. SpriteKit - it's like SceneKit, but for 2D graphics. It has support for displaying video with SKVideoNode . You only need to put SKVideoNode in SKScene, and SKScene in SCNMaterialProperty, and everything will be ready.

But by exporting the resulting 3D scene and opening it somewhere else, we will see a black square. Having rummaged in the scn-file, I found the reason. It turns out that when saving a video clip, it does not save the video URL. It would seem that you take and rule. But not everything is so simple: the scn-file is the so-called binary plist, which contains the result of the work of NSKeyedArchiver. And the material, which is the SpriteKit's scene, is the same binary plist, which, it turns out, already lies inside another binary plist! It’s good that there are only two nesting levels.

Well, now we are going to even the effect, and the tool that allows you to create any kind of effects. These are shader modifiers.

Before you modify something, you need to understand what we are modifying. A shader by definition is a program for the GPU that is run for each vertex and for each pixel. Thus, a shader is a program that determines how an object looks on the screen.

Well, the modifier shader allows you to change the results of the work of standard shaders to GLSL or Metal Shading Language. They are also available in a visual editor, which allows you to see changes in the modifier in real time.

Fur and Parallax Mapping

Using the modifier shader, you can create sophisticated visual effects. Here, for example, a couple of the most famous effects: Fur and Parallax Mapping .

#pragma arguments
texture2d bg;
texture2d height;
float depth;
float layers;
#pragma transparent#pragma body
constexpr sampler sm = sampler(filter::linear, s_address::repeat, t_address::repeat);
float3 bitangent = cross(_surface.tangent, _surface.normal);
float2 direction = float2(-dot(_surface.view.rgb, _surface.tangent), dot(_surface.view.rgb, _surface.bitangent));
_output.color.rgba = float4(0);
for(int i = 0; i < int(floor(layers)); i++) {
    float coeff = float(i) / floor(layers);
    float2 defaultCoords = _surface.diffuseTexcoord + direction * (1 - coeff) * depth;
    float2 adjustment = float2(scn_frame.sinTime + defaultCoords.x, scn_frame.cosTime) * depth * coeff * 0.1;
    float2 coords = defaultCoords + adjustment;
    _output.color.rgb += bg.sample(sm, coords).rgb * coeff * (height.sample(sm, coords).r + 0.1) * (1.0 - coeff);
    _output.color.a += (height.sample(sm, coords).r + 0.1) * (1.0 - coeff);
}
return _output;

Ray Casting with caustics in real time

What is even more interesting, no one bothers to completely throw out the results of their work and write your renderer. For example, you can try to implement Ray Casting in shaders. And it all works fast enough to provide 30 FPS even on such complex calculations. But this is a topic for a separate report. Come to Mobius !

Nightmare!

I do not like to blink, because the closed eyelids abruptly load the GPU for the BDPT due to lack of lighting.
- John Carmack

So, we have a bunch of objects with cool effects. Now it remains to learn how to write them. To do this, we turn to a more complex topic: how we learned to record video directly from SceneKit without an external UI and how we optimized this record dozens of times.

Let's first turn to the simplest solution: ReplayKit . Find out why it does not fit. Generally speaking, this solution allows you to create a screen entry in a few lines of code and save it through the system preview. But. It has a big minus - it records everything, the whole UI, including all the buttons on the screen. It was our first decision, but for obvious reasons it was impossible to let it go in production: users had to share the videotape, and share it not from the system preview.

We found ourselves in a situation where the solution had to be written from scratch. Quite from scratch. So, let's see how you can create your own video in iOS and record your frames there. It's pretty simple:

Recording process

You need to create an entity that will record files - AVAssetWriter , add a video stream to it - AVAssetWriterInput , and create an adapter for this stream that will convert our pixel buffer to the format required by the stream - AVAssetWriterPixelBufferAdaptor .

Just in case, I remind you that a pixel buffer is an entity that is a piece of memory where data for pixels are somehow written. In essence, this is a low-level view of a picture.

But how to get this pixelbuffer? The solution is simple. SCNView has a great .snapshot () function that returns a UIImage. We just need to create a pixel buffer from this UIImage.

var unsafePixelBuffer: CVPixelBuffer?
CVPixelBufferPoolCreatePixelBuffer(NULL, self.pixelBufferPool, &unsafePixelBuffer)
guard let pixelBuffer = maybePixelBuffer else { return }
    CVPixelBufferLockBaseAddress(pixelBuffer, 0)
    let data = CVPixelBufferGetBaseAddress(pixelBuffer)
    let rgbColorSpace = CGColorSpaceCreateDeviceRGB()
    let bitmapInfo = CGBitmapInfo(rawValue: CGBitmapInfo.byteOrder32Little.rawValue | CGImageAlphaInfo.premultipliedFirst.rawValue)
    let rowBytes = NSUInteger(CVPixelBufferGetBytesPerRow(pixelBuffer))
    let context = CGContext(
        data: data, 
        width: image.width, 
        height: image.height, 
        bitsPerComponent: 8, 
        bytesPerRow: CVPixelBufferGetBytesPerRow(pixelBuffer), 
        space: rgbColorSpace, 
        bitmapInfo: bitmapInfo.rawValue
    )
    context?.draw(image, in: CGRect(x: 0, y: 0, width: image.width, height: image.height))
    CVPixelBufferUnlockBaseAddress(pixelBuffer, 0)
    self.appendPixelBuffer(pixelBuffer, withPresentationTime: presentationTime)

We just allocate memory space, describe the format of these pixels, block the buffer for the change, get the memory address, create a context at the received address, where we describe how the pixels are packed, how many lines are in the picture and what color space we use. Then we copy there the pixels from the UIImage, knowing the final format, and remove the lock for the change.

Now you need to do so every frame. To do this, we create a display link that will call a callback on each frame, where we, in turn, will call the snapshot method and create a pixel buffer from the image. It's simple!

And no. Such a solution even on powerful phones causes terrible lags and drawdowns of FPS. Let's do the optimization.

Suppose we do not need 60 FPS. We will even be pleased with the 25th. But what is the easiest way to achieve this result? Of course, you just need to bring all this to the background thread. Moreover, according to the developers, this function is thread-safe.

Hmm, lag has become less, but the video has ceased to be recorded ...

It's simple. As they say, if you have a problem, and you will solve it with several threads, you will have 2 problems.

If you try to record a pixelbuffer with a timestamp lower than the last recorded one, then the entire video will be invalid.

Let's not then write down the new buffer until the previous entry ends.

Hmm, it got much better. But still, why did lags appear from the beginning?

It turns out that the .snapshot () function , with which we get an image from the screen, creates a new renderer for each call, draws a frame from scratch and returns it, and not the image that is on the screen. This leads to funny effects. For example, physical simulation occurs twice as fast.

But wait - why are we trying to render a new frame every time? Surely somewhere you can find the buffer that is displayed on the screen. And indeed, there is access to such a buffer, but it is quite non-trivial. We need to get CAMetalDrawable from Metal .

Unfortunately, getting from Metal directly from SCNView is not so easy for a fairly understandable reason - you can choose the type of API in SceneKit yourself, but if you look under the hood and look at the layer , you can see what it is like in the case of Metal, CAMetalLayer .

But even here failure awaits us: in CAMetalLayer, the only way to interact with a view is the function nextDrawable, which returns not occupied by CAMetalDrawable. It is assumed that you will write data to it and call the present function on it, which will display it on the screen.

The solution actually exists. The fact is that after disappearing from the screen, the buffer is not deployed, but only placed back into the pool. Indeed, why allocate memory each time, if two or three buffers are enough: one is shown on the screen, the second for rendering and the third, for example, for post-processing, if you have one.

It turns out that after displaying the buffer, the data from it will not disappear anywhere and you can safely and securely access it.

And if we, as a successor, begin in response to each call of nextDrawable () to save it, we get almost what we need. The problem is that the saved CAMetalDrawable is the one in which the image is being drawn right now.

The jump to the real solution is very simple - we keep both the current Drawable and the previous one.

And here it is, ready - direct memory access through CAMetalDrawable.

var unsafePixelBuffer: CVPixelBuffer?
CVPixelBufferPoolCreatePixelBuffer(NULL, self.pixelBufferPool, &unsafePixelBuffer)
guard let pixelBuffer = maybePixelBuffer else { return }
CVPixelBufferLockBaseAddress(pixelBuffer, 0)
let data = CVPixelBufferGetBaseAddress(pixelBuffer)
let width: NSUInteger = lastDrawable.texture.width
let height: NSUInteger = lastDrawable.texture.height
let rowBytes: NSUInteger = NSUInteger(CVPixelBufferGetBytesPerRow(pixelBuffer)
lastDrawable.texture.getBytes(
data, 
bytesPerRow: rowBytes, 
fromRegion: MTLRegionMake2D(0, 0, width, height), 
mipmapLevel: 0
)
CVPixelBufferUnlockBaseAddress(pixelBuffer, 0)
self.appendPixelBuffer(pixelBuffer, withPresentationTime: presentationTime)

So, now we do not create a context and draw a UIImage in it, but copy one piece of memory into another. The question arises: what about the pixel format? ..

It does not coincide with deviceColorSpace ... And it does not coincide with frequently used color spaces ...

This is exactly the point where the author of one of the public hearths broke, who performs the same task. All the rest did not even get here.

Well, all these tricks - for the eerie filter?

Well, I do not! In the article about ARKit you can find a mention of the fact that the image from the camera does not use the standard color space, but the extended one. And even presented the matrix of transformation of color space. But why engage in transformation if you can try writing directly in this format? It remains to find out which format of the 60 available ...

And then I took up the search. I recorded three videos in different streams with different formats, replacing them with each recording.

As a result, at about the fortieth format, we get its name. It turns out that this is none other than kCVPixelFormatType_30RGBLEPackedWideGamut . How could I not guess?

But my joy continued until the first tester. I had no words. How? I just spent a lot of time searching for the right format. It's good that the problem was localized quickly - the bug was reproduced stably and only on 6s and 6s Plus. Almost immediately after that, I remembered that wide-gamut-enabled displays started being installed only in the seventh iPhones.

Having changed wide-gamut to good old 32RGBA, I get a working record! It remains to understand how to determine that the device supports wide-gamut. There are still iPads with different types of display, and I thought that for sure you can get an ENUM type of display from the system. Having rummaged in documentation, I found it ≈ it is displayGamut in UITraitCollection .

When I gave the testers an assembly, I received good news from them - everything worked without any lags even on old devices!

As a conclusion, I want to tell you - do 3D graphics! In our application, for which augmented reality is not the main use case, people over the weekend covered more than 2,000 kilometers, watched more than 3,000 objects and recorded more than 1,000 videos with them! Imagine what you can do if you do it yourself.

Tags: