Using Pixel Bender filters on the example of creating an application for VKontakte

This article was written to continue the topic covered here , at the request of the user namata . The article will focus on the use of ActionScript and Pixel Bender filters in your application, and on those pitfalls that can be set aside in this matter. Flash gurus and others like them can go and have a cup of tea. They are unlikely to be interested in this article. But for those who are just getting acquainted with these things, this article will help you not to step on the rake once again.



So! Where to start? .. Well, it’s probably better to start with the question: “Why for VKontakte?” The answer is simple: I wanted to figure out their API :) In fact, this is not important. It’s just that with applications hosted on third-party servers, the situation is somewhat different than with applications hosted on your site. But first things first.

In this article I will not dwell on the issues of writing the filters themselves. We will assume that you already have them. Well, at least one ... If you still don’t, then here and there are a few examples. What to do with them when you already have a filter file with the extension .pbj on hand?

There are at least two ways to connect Pixel Bender filters to your Flash / Flex application. The first method involves embedding the filter using meta tags:

[Embed(source="SomeFilter.pbj", mimeType="application/octet-stream")]
private var _someFilter:Class;


In this case, further use of the filter will occur in approximately the following scenario:

var shader:Shader = new Shader(new _someFilter() as ByteArray);
var shaderFilter:ShaderFilter = new ShaderFilter(shader);

var bitmap:Bitmap = new Bitmap(someBitmapData);
bitmap.filters = [shaderFilter];


The filters property is an array of filters. But simply adding a new filter to the end of the array (for example, using the push ([...]) method) will not entail the use of this filter. You must first add a filter to the array, and then reassign this array to the filters property.

The second method involves loading filters at runtime:

var urlLoader:URLLoader = new URLLoader();
urlLoader.dataFormat = URLLoaderDataFormat.BINARY;
urlLoader.addEventListener(Event.COMPLETE, onLoadComplete);
urlLoader.load(new URLRequest("SomeShader.pbj"));
var shader:Shader;

function onLoadComplete(event:Event):void {
shader = new Shader();
shader.byteCode = loader.data;

var shaderFilter:ShaderFilter = new ShaderFilter(shader);
var bitmap:Bitmap = new Bitmap(someBitmapData);
bitmap.filters = [shaderFilter];
}


In general, after loading (or embedding) the filter, it doesn’t matter what the filter is: written in Pixel Bender or the basic language filter. Their further use is broadly similar.

So what about the stones?

Now let's talk about the features of using these filters for your application.

Commercial break : The
application that I developed is a small photo editor that allows you to change such characteristics of the image as brightness, contrast, saturation and lightening. You can change the values ​​of all image channels, as well as each channel individually. It is also possible to set the threshold values ​​of the filter.

Album selection: Photo editing: The first pitfall I spoke about was that we can get images (in this case, user photos) using the API. But we can’t access the image data :) What am I talking about? Here's what:









Suppose we have a certain variable in which we stuffed a string containing the path to the photo of interest to us on the server (we got this path as a result of a request to the server). Next, using the Image class from the Flex framework, we write something like this:

[Bindable]
private var _photoSrc:String = "path/to/image";



Further, somewhere in the code (in my case, this happened in the event handler for dragging the slider slider), we write the following:

curPhoto.filters = [shaderFilter];


The filter seems to be applied -> the picture on the screen has become a little different -> we are satisfied -> you can upload it to the VKontakte server.

But hey wait. In this case, the exact image will be sent to the server as on the original. Is the filter screwed up? Not at all.

Consider another scenario.

Suppose you used the Loader class to load an image. When using this method, there are two key points:

1. The load () method of an object of the Loader class should generally have the following signature:

loader.load(new URLRequest(this._photoSrc), new LoaderContext(true));


The second argument to the method should be an object of the LoaderContext class whose first argument will be true. As a result of this, Flash Player will try to download the security policy file before starting to download the image file itself (use case Security.allowDomain ("*"); in this case it will not feed).

2. In the event handler Event.COMPLETE we should have the following:

var bitmapData:BitmapData = new BitmapData((event.currentTarget as LoaderInfo).content.width, (event.currentTarget as LoaderInfo).content.height);
bitmapData.draw(this.loader);

this.bitmap = new Bitmap(bitmapData);
this.curPhoto.source = this.bitmap;
this.bitmap.filters = [shaderFilter];


The important thing is that we should not pass this.loader.content to the draw () method, but this.loader object itself. Why is this so, and is it a bug or a feature? I honestly did not understand. But it works, and all other options violate the security policy of the sandbox.

So, the bitmap object is in our pocket, you can add it to the display list, apply filters to it and upload it to the server. Are stories the end? Not really.

The second pitfall is that not all filters are equally useful.applying filters is not so simple. The fact is that when we apply a filter using the filters property, we do not apply it to the data of this image itself, but only to the display object of this image. Let me remind you that the BitmapData class contains the actual image data, and the Bitmap class contains a display object that can be included in the display list. As you recall, we assigned our filter to the filters property of the Bitmap object, which means that the BitmapData object itself (contained in the bitmapData property of this.bitmap object) remained unchanged (the same thing basically happened when we tried to load to the server an instance of the Image object in the case described above).

It would seem that the solution is obvious - you need to apply the applyFilter ([...]) method to some intermediate BitmapData object (in this case, the first argument to the method would be the BitmapData object from the this.bitmap.bitmapData property). But in my case, when support for changing several properties is implemented, the application of each subsequent filter overwrites all changes of the previous filter. Those. after any manipulations, only changes from the application of the last filter were visible. Unfortunately, it is impossible to use not one filter, but an array of filters in this method.

How to be?

The solution was found somewhat unusual. Remember our Image class object? Here he came in handy. Here is a sample code:

var bitmapdata:BitmapData = new BitmapData((curPhoto.content as Bitmap).width, (curPhoto.content as Bitmap).height);
bitmapdata.draw(curPhoto.content as Bitmap);

var jpgEncoder:JPGEncoder = new JPGEncoder(85);
this.jpgStream = jpgEncoder.encode(bitmapdata);

// и дальше заливаем на сервак...


Those. we created an intermediate object of the BitmapData class and rendered the contents of our Image object in it. As we recall, earlier we assigned an instance of this.bitmap of the Bitmap class (to which we applied filter (s)) to the source property of our object of class Image. This means that in fact we did not work with the data of the image itself, applying filters, but only with its display object.

Commercial break :
To show the result of applying filters in my application, I will give a couple of screenshots (before and after applying filters).

Photos before editing: ... and after:

Some thoughts on the topic:

1. In principle, to edit the parameters that are currently in my application, one could also use a convolution filter of the ActionScript language (ConvolutionFilter). As far as I can tell, it would be enough. But I just wanted to better deal with Pixel Bender'om :)). Pixel Bender is more suitable for creating complex filters. It is easier and more understandable to work with the image (more precisely, with its pixels). When using a convolution filter, you have to work with matrices. In general, this is a personal matter for everyone.

2. Pixel Bender filters are better to embed in a project than to load at runtime. They are lightweight enough (none of my filters weighed more than one kilobyte). Extra code is useless.

3. The VK API is generally good. A little annoying stage of verification by the administration. Reading the questions and answers of other developers in the topic "Questions about creating applications" in the "Discussions" section of the FLASH API group, I realized that in order for your application to be approved, it needs to not only meet all the requirements, but also to have the right phase the moon (and the parade of planets would not hurt the campaign ...). Probably I was just lucky that the application was approved the first time. Heh, by the way, the application was written from scratch in a week, and they checked it for almost a week ... It would be funny a little bit more.

4. Far from pleased with the feedback from the developers. Yes, there is a FLASH API group, there is a topic where you can ask your questions about creating applications. BUT! The questions, as I understand it, are answered by people who have no relation to the site’s administration in principle (i.e., in most cases, other developers who have already gotten bumps). And further. When I was looking for the answer to one question, I understood two things:

1. The questions in this thread of the FLASH API group are repeated at best, about five times each.
2. In order not to be a dumbbell, I honestly tried to find the answer to my question in this thread. The logic was simple: someone had to deal with this before. But damn it, reaching the "twenty-some" page, I realized that I had already forgotten what I was looking for. If someone close to the site administration is reading this post now, then you should be ashamed. There is completely no search or at least some structure.
   

PS: Well, at the end of the topic I will give a link to the application itself. All suggestions / suggestions for expanding the project are welcome. If users like the application, I can implement almost everything that is in the photo editor on picnik.com , and even more (of course, within the framework of the VK API;)).