Good day, Habr!

I am a member of a small company (of two people) that makes live wallpapers for Android devices. This article will talk about the development of our applications, from relatively simple to more complex, applied technologies, tricks and resolved problems - all using concrete examples, in a (almost) chronological order. All our wallpapers are fully three-dimensional, written using OpenGL ES.

As a bonus - a small collection of shaders, use them however you want. Suggestions for their improvement or correction will be welcome - we do not claim to be a guru in this area.

“Watch” and “HTC” - the first joy

The engine used was JPCT-AE , still old, which uses OpenGL ES 1.0. Loading models came from 3DS format. There were no shaders, compressed textures, render-to-texture, and other fancy stuff here. However, here you could do without all this.

“New Year tree” - the disadvantages of OpenGL ES 1.0

Here 2 things turned out: the emulator disgustingly draws three-dimensional graphics and JPCT-AE devours a lot of memory on the texture.

The emulator of that time not only drew poorly three-dimensional graphics of OpenGL ES 1.0, but drew it through chur incorrectly and slowly. Since then, development has gone only on a real device.

The version of JPCT-AE used then allocated RAM for each texture, and the program could suddenly shut down due to lack of memory. We hope this bug has already been fixed in new versions of the engine. It is known that Android applications do not use OpenGL resources to calculate the allocated Dalvik memory, but here the engine had problems with not freeing memory after loading the texture into video memory. I had to reduce the size of the textures not because the video card could not cope with the rendering, but because they all got stuck in the memory and the program crashed.

Also, when using the then version of JPCT-AE, there was a vile bug when re-creating the OpenGL context - textures could be lost or messed up. No solution to this problem was found. Again, we hope that in the current version of JPCT-AE this bug is fixed.

In fairness, we add that the latest versions of JPCT-AE added support for OpenGL ES 2.0 and their own shaders, so that novice developers can try to use it.

“Rose” - OpenGL ES 2.0 and alpha testing

Here we switched to using pure OpenGL ES 2.0, without using any frameworks or engines.

The reason for switching to OpenGL ES 2.0 was that to display roses in OpenGL ES 1.0, you had to use alpha-blending, with sorting polygons, of course. Which would still lead to poor performance due to excessive re-rendering (depth-testing is turned off after all).

Naturally, these problems are easily solved using alpha testing, and OpenGL ES 2.0 allowed us to implement it elementarily. However, there were some difficulties with this. Alpha testing itself is implemented elementarily:

vec4 base = texture2D(sTexture, vTextureCoord);
gl_FragColor = base;
if(base.a < 0.5) { discard; }

However, this obvious implementation has some disadvantages associated with loading textures from a PNG file.
Firstly, the edges of the petals get a thin black line, which is especially noticeable on light petals:



This happens if you use the standard texture loading method, which multiplies the color by the alpha channel. An artifact can be avoided by implementing your own texture loading.

Secondly, the texture is uncompressed, it takes up a lot of video memory, and therefore it is drawn slower. Believe me, compressed textures really immediately give a performance boost. The problem here is that the only standardized texture compression algorithm for OpenGL ES 2.0 is ETC1. And it does not support alpha channel. So you have to do two textures - for color (diffuse), and transparency (opacity). Two compressed textures still occupy less memory than one uncompressed texture of the same size (and, therefore, work faster). You can also decide that for specific cases the transparency texture can be made smaller than the color texture, or vice versa.
But all these problems were resolved later, when the following live wallpapers were developed - Turbines.

“Turbines” - shaders, shaders, shaders ...

Here we began to truly use the capabilities of OpenGL ES 2.0 shaders. Of the new shaders here is fog (so far pixel-wise) and a landscape shader, through which shadows from the clouds run.

The fog was chosen linear, although they immediately tried a more realistic exponential one. The exponential fog on mobile devices slowed down so that we immediately abandoned it.

The landscape shader uses two textures - a repeating 512 * 512 grass texture and a second texture that combines the static landscape lightmap and cloud shadow texture (only 64 * 128 in size). Here, various operations are performed on the colors - the lightmap is multiplied, the shadows and fog are combined using mix ().
It turned out further that these shaders were damn unoptimized, since everything was written in the pixel shader code.

“Pumpkin” - new shaders, faster loading

Here we implemented quick loading of models in a ready-made binary format, and then VBO, and for the first time we applied texture compression ETC1 (we already spoke about compressed textures using the example of a rose).

It so happened that the new loading of models implemented in these live wallpapers brought more benefits when roses were used on wallpapers. Initially, the models were loaded from the OBJ format, which was rather slow and caused complaints from users of the “Rose”. After all, the model here is actually rather big, a little more than 1000 triangles, and the OBJ file was parsed for a long time. A utility was written that created files with data ready for transfer to glDrawArrays (). The performance gain is difficult to describe - before the rose loaded for 10 or more seconds, now loading can be described only as “instantly”.

I want to note that mobile devices greatly surprised us in their ability to draw a large number of polygons - as has just been said, 1000 triangles in a rose was not a problem for any device, and in our other tests and wallpapers we already use much more. VBO was a disappointment, however - absolutely no performance gain was noticed.

Pumpkin shader is based on some example of fabric from RenderMonkey, a little simplified and finalized.

“Melting candle” - animation, simple sorting

For these wallpapers, we have developed a simple vertex animation algorithm, with simple linear interpolation between frames. Software animation, i.e. on the CPU. It is used for two purposes - candle animation (several frames of animation) and shadow animation. Yes, there are no shadow maps here, shadows are separate models with a texture calculated in 3ds max. Shadow models have 2 frames of animation each - a normal state and a little stretched in the direction from the light source. Between these frames smooth animation is performed, simultaneously with scaling of the candle flame:



Also, the development of vertex animation allowed us to add birds to the “Turbines” wallpaper.

Well, as you can imagine, here we trained with blending and switching depth-testing. There are 2 transparent objects that need to be sorted - this is a candle flame and a feather. Instead of sorting precisely by distance to the camera, we check the current position of the camera. The camera moves in a circle, on a separate sector of which you need to draw first a flame, then a feather, on the rest - vice versa.

“Lantern Festival” - vertex shaders, drawing order

As you can see from the picture, here we will talk about the shader of water and fog. What you might not have noticed is that there is also something to tell about the sky shader.

The water shader is taken from the PowerVR example, and simplified (threw refraction, maybe somewhere simplified the calculations). Reflection is rendered into a small texture - only 256 * 256.

For the landscape, a rather complex fog shader is used: the density of the fog depends on the distance and height, and the color of the fog depends on the position of the moon (the fog is highlighted around the moon). All this was possible to implement with sufficient performance only in the vertex shader. After that, we rewrote the fog shader in Wind Turbines to vertex, this gave a noticeable increase in productivity.

Now about the sky shader. The sky is drawn from 3 layers: the actual texture of the sky with the moon, a layer of stars (stored in the alpha channel of the color texture) and moving clouds. Stars are not combined with the texture of the sky because they need more brightness. Initially, the shader was written something like this:

const float CLOUDS_COVERAGE = 12.0;
vec4 base = texture2D(sTexture, vTextureCoord);
vec4 clouds = texture2D(sClouds, vec2(vTextureCoord.x + fOffset, vTextureCoord.y));
float stars = (1.0 - base.a) * starsCoeff;
stars *= pow(1.0 - clouds.r, CLOUDS_COVERAGE);
base = base + stars;
gl_FragColor = base + base * clouds * 5.0;

It would seem that there is little code, but still the performance was low, something had to be optimized. As you can see, the rather thick pow () function is used to darken the stars with clouds, and additional brightness enhancement of the clouds (clouds * 5.0) is also performed. We managed to replace the darkening of the stars with clouds with a linear clamp (), and we managed to get rid of the gain (clouds * 5.0) altogether, making the texture of the clouds brighter in Photoshop. The final shader began to work a little faster:

vec4 base = texture2D(sTexture, vTextureCoord);
vec4 clouds = texture2D(sClouds, vec2(vTextureCoord.x + fOffset, vTextureCoord.y));
float stars = (1.0 - base.a) * starsCoeff;
stars *= clamp(0.75 - clouds.r, 0.0, 1.0);
base = base + stars;
gl_FragColor = base + base * clouds;

However, this performance was still lacking. Attempts to render the simplified sky into a reflection texture yielded nothing - the texture was only 256 * 256, even if you did not draw the sky at all, nothing changed. The problem turned out to be completely different - that the sky was drawn first, not the last. Those. video card resources were spent on drawing a whole hemisphere of the sky, and then still wiping it with the landscape and other objects. We carefully reviewed the rendering order in our program, built objects in the optimal order, and this achieved the necessary performance. So even when it seems to you that you have completed your OpenGL application and everything is working well, finally review your drawing order - by rearranging a couple of lines of code in places, you can improve performance quite well.

“Tulip” - sorting objects

We made these live wallpapers quite quickly - there is little modeling, no new shaders are needed, the only thing that had to be implemented was sorting.

You need to sort, of course, transparent “defocused” tulips. Each of them is a separate object, we calculate the square of the distance from it to the camera and sort by this value (the square of the distance is enough, because the value is needed solely for comparison).

The performance of these wallpapers turned out to be not so large as that of Rosa, which was quite expected - a lot of transparent polygons were added here, but they are not optimized in any way, everything is drawn that hit the screen. To achieve acceptable performance, I had to select the FOV and the number of “blurry” colors on the screen - the smaller the better.

“Jelly Bean” - glare, blending

For glare on glass and “beans” we used simplified calculations. Usually, the vector of the direction of light is involved in the calculation of the glare, and if we assume that it is equal to some simple constant like (0; 0; 1), then the calculations are very simplified - the dot () operation is thrown, etc. The result is as if the light source is in the same place as the camera, but for such a simple scene this is more than enough.

The glass shader selects the color and alpha depending on the screen normal and glare. Works together with the right blending, naturally.

“Lotus” - dragonflies and interpolation

Let's start with the shaders. The water here is the same as in the "lights", but with changed parameters. To change day and night, multiplication by ambient-color is added to all objects:

...
gl_FragColor *= ambientColor;

Animation of dragonfly wings is implemented unusually - the geometry of the wings is not animated. Such a flapping of wings cannot be conveyed by an animation of geometry. The wings here are akin to the blades of a helicopter - they move very quickly and for the naked eye merge into a single slowly shimmering, rotating plane. So we made a model of wings from several intersecting planes, and the effect of fast flapping is created by a shader, performing a simple shift of UV-coordinates.

Static wing geometry:



In these wallpapers you can observe many objects moving along smooth paths - dragonflies, butterflies, fish underwater and the camera. All of them move along splines, and bicubic interpolation based on the Catmull-Rom algorithm is implemented for smooth movement. Note that the interpolation is slightly different from the one that smooths splines in 3ds max with Smooth vertices. One hundred percent accuracy for most objects was not useful to us here. Also, the disadvantage of the applied algorithm is that for uniform motion, the segments between the spline vertices must be the same length - otherwise the movement on short segments will slow down, and on long ones it will accelerate. And for the movement of the camera, this is already important.

Note that to create a spline with even segments, you can use the “Normalize Spline” modifier in 3ds max.

But in our case, the drawback with long segments was eliminated by the way we exported animation from 3ds max to text format (in order to extract values ​​from it into an array). To export the animation, we used the “Export Animation” command, which generates an XML file (* .xaf) with the animation. In it you can find not only all the vertices of the curve along which the object is moving, but also its position for each position on the track bar. Thus, you can animate your object as you like - by any controllers, keyframe animation, paths and even all of this at the same time, and at the output get the coordinates of its position in space depending on time.

For the halo effect, the fireflies made a vertex shader that positions and scales the halo sprite:

Shader collection

You can download the shader collection here: dl.dropbox.com/u/20585920/shaders.zip

All the shaders are made in the RenderMonkey program, you can download it from AMD: developer.amd.com/resources/archive/archived-tools/gpu-tools -archive / rendermonkey-toolsuite / # download

If you own a GeForce graphics card, then RenderMonkey may not start. You will need to use the 'nvemulate' utility. Download here: developer.nvidia.com/nvemulate . Launch it and change the 'GLSL Compiler Device Support' setting to 'NV40':

UPD: Where to start yourself

Want to write your live wallpaper with OpenGL? Not sure what to do with GLWallpaperService and other smarts? Everything has been written for a long time, take the code of examples and change to your needs:
Google OpenGL ES 2.0 example : code.google.com/p/gdc2011-android-opengl
live wallpaper example using OpenGL: github.com/markfguerra/GLWallpaperService