Android - live wallpapers on OpenGL ES 2.0 with simplified DOF effect
The idea for creating the application was a video, in turn, inspired by the design of Deus Ex: Human Revolution, namely its elements with glass fragments. In a couple of days, the final vision of the scene has matured and as a result, the first version of the finished application was created over the weekend.
The scene is extremely simple. It consists of actually smoothly rotating glass fragments hovering in the air. The fragments are divided into 2 groups - located near and far. Objects in the distance are blurry to give the scene extra volume. There is also a blurry background of arbitrary color. The camera moves left and right according to the movement of the finger across the screen, and thus the fragments move.
Technically, this is implemented as follows: Glass fragments are placed along the cylinder around the chamber. Fragments located beyond a certain distance are drawn into a separate texture and blurred. Together with them, the background is also drawn. At first, blurry objects are drawn in the distance, and fragments closer to the camera are already drawn on top of them.
In such a simple way, we realized the simplified effect of depth-of-field (DOF), which is quite enough for a given scene. A full-fledged DOF implementation is somewhat more complicated and more resource-intensive. To do this, it would be necessary to render a map of the depth of the scene, the finished scene into separate textures, and it again with blur. And then draw on the screen at the same time a blurry and clear scene, mixing them according to the depth map and the focus parameters of the camera.
Since all objects in the scene are transparent, all rendering is done with different blending modes. At the same time, the depth buffer is not written so that transparent glasses do not cut off objects behind them. Since the glasses themselves are few, this does not cause too much repeated rendering of pixels. To create highlights and reflections, fragments objects are drawn with a 128x128 cubemap.
1. Scrub FBO with glClearColor desired color.
2. A mask for the entire size of the frame buffer is drawn on top. Thus, we get decorative colored blurry spots for the background instead of a solid color.
3. Then the windows are drawn for the background. The resolution is 256x256, the image is pretty pixelated.
4. Blur the whole background. The low resolution of the background is almost imperceptible.
1. Screen cleaning.
2. Background drawing.
3. Rendering foreground glasses.
This rendering is done without writing to the depth buffer, since all objects are transparent.
Blur is implemented by sequentially rendering an image between two frame buffers with a special shader. The shader can do horizontal or vertical blur, this is determined by the parameters. This blurring method is called ping-pong rendering. Its essence is that first the texture from frame buffer A is drawn with horizontal blur to frame buffer B, and then vice versa from B to A, but with vertical blur. This procedure can be repeated with the necessary number of iterations to achieve the desired blurring quality of the original image. An example of the implementation of this postprocessing effect was taken long ago from some bloom example, unfortunately I can’t find the link.
It is noteworthy that modern phones and tablets (and even very old devices, too) can manage to carry out not just one, but several iterations of blur quickly enough. In practice, it turned out that Nexus 10 produces stable 50-40 fps even with 6-8 passes of 256x256 texture blur, and one pass is full - horizontal + vertical blur.
Choosing a fairly compromise ratio of texture resolution, number of passes and blur quality, we settled on three iterations and a resolution of 256x256.
In a previous article, I threw a stone into the nVidia garden. This is not because I just don't like nVidia - I just like any other hardware manufacturers that provide buggy drivers for their GPUs. For example, when developing the described live wallpapers, they encountered a problem on the Nexus 10. The problem is incorrect rendering in the texture, and this only manifests itself when the orientation of the device changes. How the orientation of the tablet can affect the rendering of the texture remains a mystery to us, but it is a fact.
First, in order to make sure that I just missed some nuance when initializing the context, I wrote a question on Stack Overflow: stackoverflow.com/questions/17403197/nexus-10-render-to-external-rendertarget-works-only-in- landscapeAnd here it is worth praising the ARM staff for their work. support. After a couple of days, I received a letter from an ARM engineer in which he suggested giving a request about this bug to the Mali Developer Center forum. I prepared a simple test application and described the steps for reproducing the error: forums.arm.com/index.php?/topic/16894-nexus-10-render-to-external-rendertarget-works-only-in-landscape/page__gopid__41612. And after only 4 (!) Days I received an answer that there really is a bug in the current version of the video driver for Nexus 10. The most interesting thing is that ARM suggested workaround to solve my problem, which miraculously helped - you just need to call glViewport ( ) after glBindFramebuffer (). For such work of those. they should put a monument to ARM support in life - an ARM employee was not too lazy to find my e-mail (and he is not listed on Stack Overflow), and those engineers. ARM support found and resolved the problem faster than I even expected.
Anyone interested in the quality of Android on Nexus 10, please vote for the corresponding bug in the Google tracker: code.google.com/p/android/issues/detail?id=57391
You can download the program from Google Play at: play.google.com/store/apps/details?id=org.androidworks.livewallpaperglass
The described method of the simplified DOF effect can be applied not only to a scene with the same objects, as in our application, but also in any other cases where you can separate the main scene from the background.
Scene building
The scene is extremely simple. It consists of actually smoothly rotating glass fragments hovering in the air. The fragments are divided into 2 groups - located near and far. Objects in the distance are blurry to give the scene extra volume. There is also a blurry background of arbitrary color. The camera moves left and right according to the movement of the finger across the screen, and thus the fragments move.
Technically, this is implemented as follows: Glass fragments are placed along the cylinder around the chamber. Fragments located beyond a certain distance are drawn into a separate texture and blurred. Together with them, the background is also drawn. At first, blurry objects are drawn in the distance, and fragments closer to the camera are already drawn on top of them.
In such a simple way, we realized the simplified effect of depth-of-field (DOF), which is quite enough for a given scene. A full-fledged DOF implementation is somewhat more complicated and more resource-intensive. To do this, it would be necessary to render a map of the depth of the scene, the finished scene into separate textures, and it again with blur. And then draw on the screen at the same time a blurry and clear scene, mixing them according to the depth map and the focus parameters of the camera.
Implementation
Since all objects in the scene are transparent, all rendering is done with different blending modes. At the same time, the depth buffer is not written so that transparent glasses do not cut off objects behind them. Since the glasses themselves are few, this does not cause too much repeated rendering of pixels. To create highlights and reflections, fragments objects are drawn with a 128x128 cubemap.
Background rendering order:
1. Scrub FBO with glClearColor desired color.
2. A mask for the entire size of the frame buffer is drawn on top. Thus, we get decorative colored blurry spots for the background instead of a solid color.
3. Then the windows are drawn for the background. The resolution is 256x256, the image is pretty pixelated.
4. Blur the whole background. The low resolution of the background is almost imperceptible.
Drawing the main scene and arranging two plans:
1. Screen cleaning.
2. Background drawing.
3. Rendering foreground glasses.
This rendering is done without writing to the depth buffer, since all objects are transparent.
Background blur
Blur is implemented by sequentially rendering an image between two frame buffers with a special shader. The shader can do horizontal or vertical blur, this is determined by the parameters. This blurring method is called ping-pong rendering. Its essence is that first the texture from frame buffer A is drawn with horizontal blur to frame buffer B, and then vice versa from B to A, but with vertical blur. This procedure can be repeated with the necessary number of iterations to achieve the desired blurring quality of the original image. An example of the implementation of this postprocessing effect was taken long ago from some bloom example, unfortunately I can’t find the link.
It is noteworthy that modern phones and tablets (and even very old devices, too) can manage to carry out not just one, but several iterations of blur quickly enough. In practice, it turned out that Nexus 10 produces stable 50-40 fps even with 6-8 passes of 256x256 texture blur, and one pass is full - horizontal + vertical blur.
Choosing a fairly compromise ratio of texture resolution, number of passes and blur quality, we settled on three iterations and a resolution of 256x256.
Mali
In a previous article, I threw a stone into the nVidia garden. This is not because I just don't like nVidia - I just like any other hardware manufacturers that provide buggy drivers for their GPUs. For example, when developing the described live wallpapers, they encountered a problem on the Nexus 10. The problem is incorrect rendering in the texture, and this only manifests itself when the orientation of the device changes. How the orientation of the tablet can affect the rendering of the texture remains a mystery to us, but it is a fact.
First, in order to make sure that I just missed some nuance when initializing the context, I wrote a question on Stack Overflow: stackoverflow.com/questions/17403197/nexus-10-render-to-external-rendertarget-works-only-in- landscapeAnd here it is worth praising the ARM staff for their work. support. After a couple of days, I received a letter from an ARM engineer in which he suggested giving a request about this bug to the Mali Developer Center forum. I prepared a simple test application and described the steps for reproducing the error: forums.arm.com/index.php?/topic/16894-nexus-10-render-to-external-rendertarget-works-only-in-landscape/page__gopid__41612. And after only 4 (!) Days I received an answer that there really is a bug in the current version of the video driver for Nexus 10. The most interesting thing is that ARM suggested workaround to solve my problem, which miraculously helped - you just need to call glViewport ( ) after glBindFramebuffer (). For such work of those. they should put a monument to ARM support in life - an ARM employee was not too lazy to find my e-mail (and he is not listed on Stack Overflow), and those engineers. ARM support found and resolved the problem faster than I even expected.
Anyone interested in the quality of Android on Nexus 10, please vote for the corresponding bug in the Google tracker: code.google.com/p/android/issues/detail?id=57391
Result
You can download the program from Google Play at: play.google.com/store/apps/details?id=org.androidworks.livewallpaperglass
The described method of the simplified DOF effect can be applied not only to a scene with the same objects, as in our application, but also in any other cases where you can separate the main scene from the background.