January 6, 2013 at 13:18Minicomputer from a router with OpenWRT: writing a framebuffer driver
Good afternoon, dear Khabrovites. So we come to the most interesting and important part of my series of articles about turning a small router into a minicomputer - now we will be developing a real framebuffer driver that will allow you to run different graphic applications on the router. So that enthusiasm does not fade away, here is a video of one of these applications - I think most will recognize this magnificent old quest:
In case you skipped the previous parts, here are the links:
1 - Minicomputer from a router with OpenWRT: developing a USB video card
2 - Minicomputer from a router with OpenWRT: writing USB class-driver for Linux
So, let's get to work.
Our hardware has not changed again (although we will definitely change something in the firmware in the next article), so let's start with a review of what we have to do.
Despite the fact that our driver from the previous article already knows how to display graphics, it is very difficult to use it fully (for example, display a console or launch a graphical application). The fact is that such applications require a well-defined, standardized interface, the so-called framebuffer. When registering the framebuffer in the system, we will need to fill out several specialized descriptor structures, this time much more specific than just the abstract “file operations” that we completed the last time. Operations will also be there, but besides them there will be special callbacks, such as fb_imageblit(called when someone wants to transfer a block of bits with an image to a specific location on the screen), fb_copyarea (a similar transfer, but the block of bits is taken not from an external buffer but from video memory), etc. In addition, there will be structures with a description of the screen resolution, color bitness and how color components are located in this “bitness”.
The first thing you need to realize: we have a somewhat non-standard situation, when compared with PC video cards (although it’s quite usual for embeddable devices) - our device does not have video memory as such, which we could access - more precisely, it has memory, the same GRAM hidden in the bowels of the display, but we only have access to it through a “window” of 16 bits wide. There is also not enough memory on board to store the entire frame buffer there.
Fortunately, Linux has a special approach for this - we have already written functions with the prefix " sys_ ", for example, " sys_imageblit ", which implement the necessary framebuffer functionality, working with the system’s main memory area as a frame buffer.
That is, if the frame buffer we have is located on the video card and we have hardware support for such operations, we simply kick our piece of iron in the callbacks, giving the command “execute bit block transfer” or “copy the video memory area”.
If we don’t have any of this, we allocate memory in the kernel equal in size to our frame buffer, and in the callbacks we call these same functions with the prefix " sys_ ", which perform the necessary operations in RAM.
Thus, you can get a fully working framebuffer that will not interact with hardware at all - there is already such a driver and it is called vfb, virtual framebuffer. Its source code is in the drivers / video / vfb.c .
If we add periodic data sending to such a driver to a real device, we will get a real framebuffer driver. But before we get into this, let's tune our system a bit and practice on the virtual driver, vfb .
I spent a lot of time with this part, mainly because I wrote my driver first and then tried to understand why I only had a black screen - I sinned for my mistakes in the code. Then he guessed to install the VFB driver instead, having read the contents of the memory of which he saw the same black screen. Then, at last, I realized that the problem was not with the driver, but with the fact that the kernel itself refused to output information to it, after which the problem was solved rather quickly. But first things first.
Now we will do all three points regarding the driver of the virtual framebuffer, and when we get the picture in memory, let's move on to writing our own. fbcon and the framebuffer driver can either be built statically, or both as plug-ins, or one statically, the second dynamically - this will not cause problems, fbcon will snap the framebuffer as soon as it sees it.
True, when working with vfb there is one subtlety - to activate it, you must pass the module the parameter vfb_enable = 1 , or start the kernel with the parameter " video = vfb ". It will be easier to work with the module, so we will limit ourselves to it. fbcon we will build in a kernel.
The last thing we need to do before rebuilding is to go to make menuconfig and include the fbset utility in the busybox options that will allow us to set the parameters of our framebuffer. It is located in the menu Base System - Busybox - Linux SyStem Utilities
Now you can rebuild the kernel. In build_dir / target-mips_r2_uClibc-0.9.33.2 / linux-ar71xx_generic / linux-3.6.9 / dri
vers / video /
we take away that with the extension .ko
Contrary to expectations, there will be more than one there, enabling a driver that uses functions with a prefix " sys_"activates the assembly of several modules in which these functions lie. What is interesting, in principle, does not interfere with building them statically into the kernel, leaving the driver a plug-in, but I could not do this from the menu, I had to write the corresponding patch to the Kconfig file But we will do this later, and now we just restart the router with the new firmware and transfer all the modules to it, then
go over SSH to the router and go to / etc. Open the inittab file and see something like this:
The last line just says that you need to run the software for the login (in this case, like all system binaries - part of busybox) on ttyATH0, the serial port. It is indicated (askfirst) that to activate this console, you will first need to press enter.
Add one more line:
Let's see if the kernel parameters are specified correctly through
and reboot the router.
Now, in turn, we inject everything except the vfb driver, and at the very end we write
After that, we should see words like these in dmesg: Console: switching to color frame buffer device 53x21
The size of the console will differ depending on the font selected. Set the framebuffer parameters, more similar to the parameters of our display:
This will set the visible and virtual resolution to 320x240 (most often they coincide, but in principle, you can set the virtual resolution more than the visible one, get the frame buffer larger than the output and use this for double buffering), and the color depth is 16 bits.
fbcon should respond to this by changing its resolution and message to dmesg, but if this does not happen, disconnect the console from the framebuffer and reconnect:
This is a useful pair of commands that will come in handy to us more than once - without this, you cannot unload the framebuffer driver, because he will be busy with the console.
It doesn’t hurt to connect a keyboard to the router - you can blindly enter the clear and dmesg commands to be sure that there is something on the virtual display.
After we get the “screenshot” command
And download it to the desktop. There, we open it through GIMP or any other software that can load raw graphic data in RGB565 format - we set the image size to 320x240, do not forget about the bit size, and we get a picture like this (messages about opening and closing / dev / fb0 are issued by my driver, etc. K. I didn’t take the screenshot from the virtual framebuffer. The virtual one is silent about such matters):
Have you noticed the beautiful, "hacker" green color of the console? In fact, this tells us about a mistake, or rather, about one feature that needs to be reckoned with. But we will talk about this later. Before moving on to the main action - writing your framebuffer driver - let's compare the available console fonts. To do this, I prepared six photos, three on the console and on the Midnight Commander with fonts 4x4, 6x11 and 8x8. In my opinion, the most convenient is 6x11:
4x4
6x11
8x8
4x4
6x11
8x8
To begin with - about the approach. An obvious and not very good solution would be to periodically update the entire screen - you could start a timer that would throw all the contents of the framebuffer via USB to the commands already familiar to us from the previous article. However, there is a much more correct solution, the so-called deferred io.
Its essence is simple: we only need to specify the callback function, set the time interval and register this very deferred io for our framebuffer. During registration, virtual memory will be configured so that accessing the frame buffer memory will raise an exception, which will put our callback into the queue for processing at the interval specified by us. In this case, the write operation to the memory will not be interrupted! And when the callback is called, a list of pages that have been changed will be transferred to it. Is it really very convenient? Userspace can safely write in video memory without thinking about anything and without interruption, while our callback will periodically twitch with a list of memory pages that have been changed - we will need to throw only them on the device, not the entire buffer.
Since the release of the framebuffer is not as simple as it seems at first glance (the USB device may already be missing, but the clients themselves will not let go of the framebuffer yet and it’s impossible to clear the memory at this moment), we will not do very well - write in bold letters TODO and vow promise to implement the correct cleaning and shutting down the device a little later, but for now we will write everything else to finally see the result of our actions. Normal cleaning, along with finishing the video card firmware (which will raise FPS at least twice), we will definitely consider in the next article.
Let's start with a simple one - since we will receive a list of pages to which we recorded, it is easiest to save the coordinates on the screen in advance corresponding to the top of the page and also a pointer to the corresponding memory area. Therefore, we will create a structure with these fields, not forgetting to add an atomic flag that indicates whether this page requires an update. This is necessary because the internal operations of the framebuffer, those that are performed through functions with the " sys_ " prefix, do not call our deferred io handler , so we will need to manually calculate which pages were accessed and mark them as subject to update.
Everything is transparent here, the only thing is that we store the length, because the last page may be incomplete - we don’t need to send extra data to the display.
We’ll declare several defines related to the size of the display - the number of pixels in the page, the number of pages in the framebuffer, etc.
PAGE_SIZE is defined for us in the kernel source.
We declare the required callbacks of the framebuffer operations and the callback handler of our deferred io .
Next is a structure with immutable display information.
Everything is simple too - we set the string ID of our display, the type of pixels - others do not interest us, we have the most common bitmap, we have truecolor color, at least it’s close to it and certainly not monochrome and not direct color.
Next comes a more interesting structure with (potentially) variable information. But since we will not realize the possibility of changing the permit, for us it will be as constant as in the previous structure.
Here we see the visible and virtual resolutions that are already familiar with fbset , the physical dimensions of the display in millimeters (you can set, in principle, any, I set the same as its resolution), the bitness of the image, and - important structures - descriptors of how color components are in bytes. In them, the first value is the offset, the second is the length in bits and the third is the flag, “meaning bit on the right”.
Last to declare deferred I / O, deferred io .
Choosing a period value, field .delay, the task is, for the most part, empirical - if you select too small a value, the update will occur less often than the equipment allows, if you select too large, the queue of deferred work will be clogged. Our display is currently more than slow-moving, and this is determined entirely by USB, and not the output to the screen. In the implementation of the first article, a complete redrawing of the screen is possible with a frequency of no higher than 3.6 FPS. But do not despair - firstly, we will not always redraw the entire screen, and secondly, in the next article, I will show how to squeeze the maximum out of our available iron, so that the FPS will jump to ~ 8 - taking into account incomplete redrawing we get a completely usable device, as in the video at the beginning of the article. These 9 FPS, by the way, are the physical limit, which we can get on Full Speed USB when transferring raw video data (without compression). This becomes apparent if we recall the FS USB transfer rate limit of 12 Mb / s. Our frame occupies 320 * 240 * 16 = 1,228,800 bits. Thus, the ideal, spherical in a vacuum frame rate will be no higher than 9.8 FPS. We drop our headers from here, losses in our driver, losses in the host controller driver, losses in the driver on STMke - and we get a real limit of 8-9 FPS, which is quite good to achieve. But we will do this in the next article, and now remember that our frequency is about 3.5 FPS, and, according to my measurements, the period about two times longer, that is, 6-7 hertz, turned out to be optimal. We set this, using a kernel predefined in the source code Our frame occupies 320 * 240 * 16 = 1,228,800 bits. Thus, the ideal, spherical in a vacuum frame rate will be no higher than 9.8 FPS. We drop our headers from here, losses in our driver, losses in the host controller driver, losses in the driver on STMke - and we get a real limit of 8-9 FPS, which is quite good to achieve. But we will do this in the next article, and now remember that our frequency is about 3.5 FPS, and, according to my measurements, the period about two times longer, that is, 6-7 hertz, turned out to be optimal. We set this, using a kernel predefined in the source code Our frame occupies 320 * 240 * 16 = 1,228,800 bits. Thus, the ideal, spherical in a vacuum frame rate will be no higher than 9.8 FPS. We drop our headers from here, losses in our driver, losses in the host controller driver, losses in the driver on STMke - and we get a real limit of 8-9 FPS, which is quite good to achieve. But we will do this in the next article, and now remember that our frequency is about 3.5 FPS, and, according to my measurements, the period about two times longer, that is, 6-7 hertz, turned out to be optimal. We set this, using a kernel predefined in the source code which is quite good to achieve. But we will do this in the next article, and now remember that our frequency is about 3.5 FPS, and, according to my measurements, the period about two times longer, that is, 6-7 hertz, turned out to be optimal. We set this, using a kernel predefined in the source code which is quite good to achieve. But we will do this in the next article, and now remember that our frequency is about 3.5 FPS, and, according to my measurements, the period about two times longer, that is, 6-7 hertz, turned out to be optimal. We set this, using a kernel predefined in the source codeHz . By the way, you should pay attention - despite the name, this macro does not determine the frequency of 1 Hz, but the corresponding period (in quantum quanta), therefore, to obtain a frequency of 6 Hz it should not be multiplied, but divided by six.
Finally, fill the structure with callbacks of operations on the framebuffer.
For reading, we immediately write the callback of those that are " sys_ ", we have nothing to do there. In all the rest, we will still need to mark the corresponding pages for updating, so we indicate our own.
The descriptor structure of our device will hardly change from the time of the previous article, we will describe it.
An array of our video memory page descriptors has been added to it. Actually, it is not recommended to statically place large amounts of data in the kernel, but the structure itself has come out small, and in total there are only 38, so in order not to fool with extra pointers, let us leave this array static, 760 bytes or so will master.
The last field, pseudo_palette, is the space under the pseudo-palette that fbcon requires . It is filled in the .fb_setcolreg callback , without which fbcon refuses to work. In all the firewood that I saw, this callback looks copy-pasted from the example framebuffer file from the kernel sources, so we won’t invent a bicycle either, especially since besides fbconnobody seems to use this. We’ll start with him.
This, as I said, is the standard code for such a callback, including the CNVT_TOHW macro, which is used to obtain the values of color components. It can also drag from the driver to the driver - it is not clear why he was in the end not introduce in the main header file fb.h .
The task of this callback is to fill in a 16-color pseudo-palette, which will be accessed by the already mentioned console driver.
Now we’ll declare a small function to which we will transfer, in fact, the rectangle in the area of which the video memory was affected. The function will calculate which pages of video memory are affected by this rectangle, put the “update required” flag on them and schedule the execution of the same callback that is called in case of deferred io. After that, all the operation callbacks will be reduced to calling the sys_ functions and the function we wrote, which we will call touch, similar to the command in Linux.
The code, I think, is understandable - we simply calculate which pages we touched on the basis of resolution, always round up, or rather, we simply take one page from each end with a margin.
Now we describe all the other callbacks - they become very simple:
Now we will finally describe our callback from deferred io , which will send information to the display. It will largely coincide with the .write callback from the previous article. We will write in the display on the pages, not forgetting to attribute, as in the previous article, to them the required title. Thanks to our videopage structure , the x, y coordinates and lengths have already been calculated, so all you need to do is just push it into the buffer and drop it over USB.
Since we can get into this callback in an honest way (via deferrd io ), or we can do it on our own initiative (by scheduling it to be performed in one of the callbacks of operations by calling display_touch), we simply go over all the pages passed to us, if any, and mark them as subject to update.
If we did not get here through deferred I / O, then the list will simply be empty.
After that, we simply go through all the pages atomically checking the need for an update and performing this very update by synchronous sending via USB. In the next article, when we finish the driver to its normal state, we will replace the synchronous sending with a more correct mechanism called the USB Request Block, or URB. It will allow the USB host to throw a request to send data and immediately return to further processing. And the fact that the URB flew (or did not fly) to the recipient will be reported to us in the interrupt. This will allow us to squeeze out a little more FPS from our system (without, however, exceeding the theoretical limit that I spoke about above).
We just had a little bit left - since we decided to do bad things and not clean ourselves, all that was left was to initialize everything in the Probe callback.
Here we first allocate memory for our device descriptor, then for the framebuffer descriptor structure , and only then for the framebuffer itself. Please note - we select through vmalloc . The difference from kmalloc is that vmalloc can reconfigure the page tables of virtual memory, and “collect in pieces” the buffer we requested. That is, for us it will look like a single block of memory, but physically it can consist of pages that are not even close to each other. kmalloc returns memory, which is physically a single block. Since we are requesting a large enough piece, vmalloc is a good practice .
We compile, insmodem, and if everything is done correctly, we see the console on the display!
In this article, we finally made this important step - from the driver of a custom device that was not used by the system, we passed into the arms of the framework, which allowed us to tell all applications and drivers that we have a real display. Yes, we did a little bad, not cleaning up our resources and not handing off the device, so pulling out USB and plugging it into a working device is not recommended yet.
But we will definitely fix this in the next article. What do we have to do?
Below is another video with an old quest (no less beloved by me than Gobliins), a few photos of a small graphical shell that can be easily run on the device and the Elinks console browser.
The first Kirandiya
Habr in a console browser (font VGA8x8)
Shell Gmenu2X
Built-in file browser in it
Settings
I have everything for now. Successful implementation and until the next article!
In case you skipped the previous parts, here are the links:
1 - Minicomputer from a router with OpenWRT: developing a USB video card
2 - Minicomputer from a router with OpenWRT: writing USB class-driver for Linux
So, let's get to work.
Introduction
Our hardware has not changed again (although we will definitely change something in the firmware in the next article), so let's start with a review of what we have to do.
Despite the fact that our driver from the previous article already knows how to display graphics, it is very difficult to use it fully (for example, display a console or launch a graphical application). The fact is that such applications require a well-defined, standardized interface, the so-called framebuffer. When registering the framebuffer in the system, we will need to fill out several specialized descriptor structures, this time much more specific than just the abstract “file operations” that we completed the last time. Operations will also be there, but besides them there will be special callbacks, such as fb_imageblit(called when someone wants to transfer a block of bits with an image to a specific location on the screen), fb_copyarea (a similar transfer, but the block of bits is taken not from an external buffer but from video memory), etc. In addition, there will be structures with a description of the screen resolution, color bitness and how color components are located in this “bitness”.
The first thing you need to realize: we have a somewhat non-standard situation, when compared with PC video cards (although it’s quite usual for embeddable devices) - our device does not have video memory as such, which we could access - more precisely, it has memory, the same GRAM hidden in the bowels of the display, but we only have access to it through a “window” of 16 bits wide. There is also not enough memory on board to store the entire frame buffer there.
Fortunately, Linux has a special approach for this - we have already written functions with the prefix " sys_ ", for example, " sys_imageblit ", which implement the necessary framebuffer functionality, working with the system’s main memory area as a frame buffer.
That is, if the frame buffer we have is located on the video card and we have hardware support for such operations, we simply kick our piece of iron in the callbacks, giving the command “execute bit block transfer” or “copy the video memory area”.
If we don’t have any of this, we allocate memory in the kernel equal in size to our frame buffer, and in the callbacks we call these same functions with the prefix " sys_ ", which perform the necessary operations in RAM.
Thus, you can get a fully working framebuffer that will not interact with hardware at all - there is already such a driver and it is called vfb, virtual framebuffer. Its source code is in the drivers / video / vfb.c .
If we add periodic data sending to such a driver to a real device, we will get a real framebuffer driver. But before we get into this, let's tune our system a bit and practice on the virtual driver, vfb .
Enable kernel graphics support
I spent a lot of time with this part, mainly because I wrote my driver first and then tried to understand why I only had a black screen - I sinned for my mistakes in the code. Then he guessed to install the VFB driver instead, having read the contents of the memory of which he saw the same black screen. Then, at last, I realized that the problem was not with the driver, but with the fact that the kernel itself refused to output information to it, after which the problem was solved rather quickly. But first things first.
- In order for us to see the console output in the framebuffer memory (well, on the screen, if it is real, and not virtual), two drivers are needed - this is, in fact, the framebuffer driver itself, which will create the device / dev / fb [x] and the console driver working on top of it is the fbcon driver
- In the kernel, accordingly, support for framebuffers , support for virtual terminals (an abstraction combining an output device + input device, display, and keyboard) should be included , support for displaying the system console on such terminals (yes, this can also be disabled, then the system console will output only to physically existing character devices like com ports), the fbcon driver itself , as well as some of the available fonts embedded in it .
- The very point that I missed at the beginning when I couldn’t understand why nothing is displayed - you need to tell the kernel that it is necessary to output the contents of the system console to the / dev / tty [x] that fbcon grabbed !
The fact is that the fbcon driver tries to capture the first available / dev / tty [x] , for example, tty0 . But the kernel doesn’t output anything there, it’s not an attached abstraction, because it does not run any application that allows you to log in to the system, nor the output of the system console.
In order to solve this problem, we must first tell the kernel that we want to see the system console on / dev / tty0(however, this is optional, if suddenly someone does not want to see the boot process and system output, then this item can be omitted), and secondly tell the Internet that there you need to run software for login
Now we will do all three points regarding the driver of the virtual framebuffer, and when we get the picture in memory, let's move on to writing our own. fbcon and the framebuffer driver can either be built statically, or both as plug-ins, or one statically, the second dynamically - this will not cause problems, fbcon will snap the framebuffer as soon as it sees it.
True, when working with vfb there is one subtlety - to activate it, you must pass the module the parameter vfb_enable = 1 , or start the kernel with the parameter " video = vfb ". It will be easier to work with the module, so we will limit ourselves to it. fbcon we will build in a kernel.
- Perform make kernel_menuconfig and go to the point of Device Drivers
- Turn on Graphics Support - Support for frame buffer devices , after which we can see the list of drivers themselves, in which we select Virtual framebuffer .
- We go back one level, go to Character devices and enable
Virtual terminal
Enable character translations in console
Support for console on virtual terminal
Support for binding and unbinding console drivers
Thanks to the last two options, we can display the system console on the virtual terminal, and then, if desired, unbind it from the framebuffer driver, which will allow it to be unloaded from memory. - We return to Graphics Support , go to the Console display driver support menu that has become available and turn on the Framebuffer Console support there , then activate the Select compiled-in fonts item and select some font there - for example, VGA 8x8 font
- We go to the main menu and pay attention to the Kernel Hacking item - if you go there, you can find an item containing kernel boot parameters near the end of the list. In general, the boot-loader passes them to the kernel, but you can supplement the parameter line with this item, or even redefine it. We will not redefine, but to supplement - we will supplement, because the bootloader passes the console = ttyATH0 parameter to it , which means that the system console is output to the serial port. Unfortunately, we cannot do this right here - this parameter will be overridden when applying the platform-specific patches, so we'll go there. Do not touch anything here, save the config and exit.
- We go to where, as we recall, platform-specific files and patches are stored - target / linux / ar71xx / . We go into generic and open the config-default file . In it we see a single line, the same parameter that we saw in the kernel setup:
CONFIG_CMDLINE = "rootfstype = squashfs, jffs2 noinitrd" we
add to the end console = tty0 and fbcon = font: <font name> , setting the font name to one of those that were selected in the kernel configuration. We get something like
CONFIG_CMDLINE = "rootfstype = squashfs, jffs2 noinitrd console = tty0 fbcon = font: ProFont6x11"
The last thing we need to do before rebuilding is to go to make menuconfig and include the fbset utility in the busybox options that will allow us to set the parameters of our framebuffer. It is located in the menu Base System - Busybox - Linux SyStem Utilities
Now you can rebuild the kernel. In build_dir / target-mips_r2_uClibc-0.9.33.2 / linux-ar71xx_generic / linux-3.6.9 / dri
vers / video /
we take away that with the extension .ko
Contrary to expectations, there will be more than one there, enabling a driver that uses functions with a prefix " sys_"activates the assembly of several modules in which these functions lie. What is interesting, in principle, does not interfere with building them statically into the kernel, leaving the driver a plug-in, but I could not do this from the menu, I had to write the corresponding patch to the Kconfig file But we will do this later, and now we just restart the router with the new firmware and transfer all the modules to it, then
go over SSH to the router and go to / etc. Open the inittab file and see something like this:
::sysinit:/etc/init.d/rcS S boot
::shutdown:/etc/init.d/rcS K shutdown
ttyATH0::askfirst:/bin/ash --login
The last line just says that you need to run the software for the login (in this case, like all system binaries - part of busybox) on ttyATH0, the serial port. It is indicated (askfirst) that to activate this console, you will first need to press enter.
Add one more line:
tty0::respawn:/bin/ash --login
Let's see if the kernel parameters are specified correctly through
cat /proc/cmdline
and reboot the router.
Now, in turn, we inject everything except the vfb driver, and at the very end we write
insmod vfb.ko vfb_enable=1
After that, we should see words like these in dmesg: Console: switching to color frame buffer device 53x21
The size of the console will differ depending on the font selected. Set the framebuffer parameters, more similar to the parameters of our display:
fbset -g 320 240 320 240 16
This will set the visible and virtual resolution to 320x240 (most often they coincide, but in principle, you can set the virtual resolution more than the visible one, get the frame buffer larger than the output and use this for double buffering), and the color depth is 16 bits.
fbcon should respond to this by changing its resolution and message to dmesg, but if this does not happen, disconnect the console from the framebuffer and reconnect:
echo "0" > /sys/class/vtconsole/vtcon1/bind
echo "1" > /sys/class/vtconsole/vtcon1/bind
This is a useful pair of commands that will come in handy to us more than once - without this, you cannot unload the framebuffer driver, because he will be busy with the console.
It doesn’t hurt to connect a keyboard to the router - you can blindly enter the clear and dmesg commands to be sure that there is something on the virtual display.
After we get the “screenshot” command
cat /dev/fb0 > scrn.raw
And download it to the desktop. There, we open it through GIMP or any other software that can load raw graphic data in RGB565 format - we set the image size to 320x240, do not forget about the bit size, and we get a picture like this (messages about opening and closing / dev / fb0 are issued by my driver, etc. K. I didn’t take the screenshot from the virtual framebuffer. The virtual one is silent about such matters):
Have you noticed the beautiful, "hacker" green color of the console? In fact, this tells us about a mistake, or rather, about one feature that needs to be reckoned with. But we will talk about this later. Before moving on to the main action - writing your framebuffer driver - let's compare the available console fonts. To do this, I prepared six photos, three on the console and on the Midnight Commander with fonts 4x4, 6x11 and 8x8. In my opinion, the most convenient is 6x11:
4x4
6x11
8x8
4x4
6x11
8x8
Writing a framebuffer driver
To begin with - about the approach. An obvious and not very good solution would be to periodically update the entire screen - you could start a timer that would throw all the contents of the framebuffer via USB to the commands already familiar to us from the previous article. However, there is a much more correct solution, the so-called deferred io.
Its essence is simple: we only need to specify the callback function, set the time interval and register this very deferred io for our framebuffer. During registration, virtual memory will be configured so that accessing the frame buffer memory will raise an exception, which will put our callback into the queue for processing at the interval specified by us. In this case, the write operation to the memory will not be interrupted! And when the callback is called, a list of pages that have been changed will be transferred to it. Is it really very convenient? Userspace can safely write in video memory without thinking about anything and without interruption, while our callback will periodically twitch with a list of memory pages that have been changed - we will need to throw only them on the device, not the entire buffer.
Since the release of the framebuffer is not as simple as it seems at first glance (the USB device may already be missing, but the clients themselves will not let go of the framebuffer yet and it’s impossible to clear the memory at this moment), we will not do very well - write in bold letters TODO and vow promise to implement the correct cleaning and shutting down the device a little later, but for now we will write everything else to finally see the result of our actions. Normal cleaning, along with finishing the video card firmware (which will raise FPS at least twice), we will definitely consider in the next article.
Let's start with a simple one - since we will receive a list of pages to which we recorded, it is easiest to save the coordinates on the screen in advance corresponding to the top of the page and also a pointer to the corresponding memory area. Therefore, we will create a structure with these fields, not forgetting to add an atomic flag that indicates whether this page requires an update. This is necessary because the internal operations of the framebuffer, those that are performed through functions with the " sys_ " prefix, do not call our deferred io handler , so we will need to manually calculate which pages were accessed and mark them as subject to update.
Memory page structure
struct videopage
{
int x, y;
unsigned char *mem;
unsigned long length;
atomic_t toUpdate;
};
Everything is transparent here, the only thing is that we store the length, because the last page may be incomplete - we don’t need to send extra data to the display.
We’ll declare several defines related to the size of the display - the number of pixels in the page, the number of pages in the framebuffer, etc.
PAGE_SIZE is defined for us in the kernel source.
Helper Defines
#define WIDTH 320
#define HEIGHT 240
#define BYTE_DEPTH 2
#define FB_SIZE WIDTH*HEIGHT*BYTE_DEPTH
#define FP_PAGE_COUNT PAGE_ALIGN(FB_SIZE)/PAGE_SIZE
#define PIXELS_IN_PAGE PAGE_SIZE/BYTE_DEPTH
We declare the required callbacks of the framebuffer operations and the callback handler of our deferred io .
Callbacks
static void display_fillrect(struct fb_info *p, const struct fb_fillrect *rect);
static void display_imageblit(struct fb_info *p, const struct fb_image *image);
static void display_copyarea(struct fb_info *p, const struct fb_copyarea *area);
static ssize_t display_write(struct fb_info *p, const char __user *buf,
size_t count, loff_t *ppos);
static int display_setcolreg(unsigned regno,
unsigned red, unsigned green, unsigned blue,
unsigned transp, struct fb_info *info);
//---------------
static void display_update(struct fb_info *info, struct list_head *pagelist);
Next is a structure with immutable display information.
Fb_fix_screeninfo structure
static struct fb_fix_screeninfo fixed_info =
{
.id = "STM32LCD",
.type = FB_TYPE_PACKED_PIXELS,
.visual = FB_VISUAL_TRUECOLOR,
.accel = FB_ACCEL_NONE,
.line_length = WIDTH * BYTE_DEPTH,
};
Everything is simple too - we set the string ID of our display, the type of pixels - others do not interest us, we have the most common bitmap, we have truecolor color, at least it’s close to it and certainly not monochrome and not direct color.
Next comes a more interesting structure with (potentially) variable information. But since we will not realize the possibility of changing the permit, for us it will be as constant as in the previous structure.
Fb_var_screeninfo structure
static struct fb_var_screeninfo var_info =
{
.xres = WIDTH,
.yres = HEIGHT,
.xres_virtual = WIDTH,
.yres_virtual = HEIGHT,
.width = WIDTH,
.height = HEIGHT,
.bits_per_pixel = 16,
.red = {11, 5, 0},
.green = {5, 6, 0},
.blue = {0, 5, 0},
.activate = FB_ACTIVATE_NOW,
.vmode = FB_VMODE_NONINTERLACED,
};
Here we see the visible and virtual resolutions that are already familiar with fbset , the physical dimensions of the display in millimeters (you can set, in principle, any, I set the same as its resolution), the bitness of the image, and - important structures - descriptors of how color components are in bytes. In them, the first value is the offset, the second is the length in bits and the third is the flag, “meaning bit on the right”.
Last to declare deferred I / O, deferred io .
Fb_deferred_io structure
static struct fb_deferred_io display_defio = {
.delay = HZ/6,
.deferred_io = &display_update,
};
Choosing a period value, field .delay, the task is, for the most part, empirical - if you select too small a value, the update will occur less often than the equipment allows, if you select too large, the queue of deferred work will be clogged. Our display is currently more than slow-moving, and this is determined entirely by USB, and not the output to the screen. In the implementation of the first article, a complete redrawing of the screen is possible with a frequency of no higher than 3.6 FPS. But do not despair - firstly, we will not always redraw the entire screen, and secondly, in the next article, I will show how to squeeze the maximum out of our available iron, so that the FPS will jump to ~ 8 - taking into account incomplete redrawing we get a completely usable device, as in the video at the beginning of the article. These 9 FPS, by the way, are the physical limit, which we can get on Full Speed USB when transferring raw video data (without compression). This becomes apparent if we recall the FS USB transfer rate limit of 12 Mb / s. Our frame occupies 320 * 240 * 16 = 1,228,800 bits. Thus, the ideal, spherical in a vacuum frame rate will be no higher than 9.8 FPS. We drop our headers from here, losses in our driver, losses in the host controller driver, losses in the driver on STMke - and we get a real limit of 8-9 FPS, which is quite good to achieve. But we will do this in the next article, and now remember that our frequency is about 3.5 FPS, and, according to my measurements, the period about two times longer, that is, 6-7 hertz, turned out to be optimal. We set this, using a kernel predefined in the source code Our frame occupies 320 * 240 * 16 = 1,228,800 bits. Thus, the ideal, spherical in a vacuum frame rate will be no higher than 9.8 FPS. We drop our headers from here, losses in our driver, losses in the host controller driver, losses in the driver on STMke - and we get a real limit of 8-9 FPS, which is quite good to achieve. But we will do this in the next article, and now remember that our frequency is about 3.5 FPS, and, according to my measurements, the period about two times longer, that is, 6-7 hertz, turned out to be optimal. We set this, using a kernel predefined in the source code Our frame occupies 320 * 240 * 16 = 1,228,800 bits. Thus, the ideal, spherical in a vacuum frame rate will be no higher than 9.8 FPS. We drop our headers from here, losses in our driver, losses in the host controller driver, losses in the driver on STMke - and we get a real limit of 8-9 FPS, which is quite good to achieve. But we will do this in the next article, and now remember that our frequency is about 3.5 FPS, and, according to my measurements, the period about two times longer, that is, 6-7 hertz, turned out to be optimal. We set this, using a kernel predefined in the source code which is quite good to achieve. But we will do this in the next article, and now remember that our frequency is about 3.5 FPS, and, according to my measurements, the period about two times longer, that is, 6-7 hertz, turned out to be optimal. We set this, using a kernel predefined in the source code which is quite good to achieve. But we will do this in the next article, and now remember that our frequency is about 3.5 FPS, and, according to my measurements, the period about two times longer, that is, 6-7 hertz, turned out to be optimal. We set this, using a kernel predefined in the source codeHz . By the way, you should pay attention - despite the name, this macro does not determine the frequency of 1 Hz, but the corresponding period (in quantum quanta), therefore, to obtain a frequency of 6 Hz it should not be multiplied, but divided by six.
Finally, fill the structure with callbacks of operations on the framebuffer.
Fb_ops structure
static struct fb_ops display_fbops = {
.owner = THIS_MODULE,
.fb_read = fb_sys_read,
.fb_write = display_write,
.fb_fillrect = display_fillrect,
.fb_copyarea = display_copyarea,
.fb_imageblit = display_imageblit,
.fb_setcolreg = display_setcolreg,
};
For reading, we immediately write the callback of those that are " sys_ ", we have nothing to do there. In all the rest, we will still need to mark the corresponding pages for updating, so we indicate our own.
The descriptor structure of our device will hardly change from the time of the previous article, we will describe it.
Usblcd descriptor structure
struct usblcd
{
struct usb_device *udev;
struct usb_interface *interface;
struct device *gdev;
struct fb_info *info;
struct usb_endpoint_descriptor *bulk_out_ep;
unsigned int bulk_out_packet_size;
struct videopage videopages[FP_PAGE_COUNT];
unsigned long pseudo_palette[17];
};
An array of our video memory page descriptors has been added to it. Actually, it is not recommended to statically place large amounts of data in the kernel, but the structure itself has come out small, and in total there are only 38, so in order not to fool with extra pointers, let us leave this array static, 760 bytes or so will master.
The last field, pseudo_palette, is the space under the pseudo-palette that fbcon requires . It is filled in the .fb_setcolreg callback , without which fbcon refuses to work. In all the firewood that I saw, this callback looks copy-pasted from the example framebuffer file from the kernel sources, so we won’t invent a bicycle either, especially since besides fbconnobody seems to use this. We’ll start with him.
Callback display_setcolreg
#define CNVT_TOHW(val,width) ((((val)<<(width))+0x7FFF-(val))>>16)
static int display_setcolreg(unsigned regno,
unsigned red, unsigned green, unsigned blue,
unsigned transp, struct fb_info *info)
{
int ret = 1;
if (info->var.grayscale)
red = green = blue = (19595 * red + 38470 * green +
7471 * blue) >> 16;
switch (info->fix.visual) {
case FB_VISUAL_TRUECOLOR:
if (regno < 16) {
u32 *pal = info->pseudo_palette;
u32 value;
red = CNVT_TOHW(red, info->var.red.length);
green = CNVT_TOHW(green, info->var.green.length);
blue = CNVT_TOHW(blue, info->var.blue.length);
transp = CNVT_TOHW(transp, info->var.transp.length);
value = (red << info->var.red.offset) |
(green << info->var.green.offset) |
(blue << info->var.blue.offset) |
(transp << info->var.transp.offset);
pal[regno] = value;
ret = 0;
}
break;
case FB_VISUAL_STATIC_PSEUDOCOLOR:
case FB_VISUAL_PSEUDOCOLOR:
break;
}
return ret;
}
This, as I said, is the standard code for such a callback, including the CNVT_TOHW macro, which is used to obtain the values of color components. It can also drag from the driver to the driver - it is not clear why he was in the end not introduce in the main header file fb.h .
The task of this callback is to fill in a 16-color pseudo-palette, which will be accessed by the already mentioned console driver.
Now we’ll declare a small function to which we will transfer, in fact, the rectangle in the area of which the video memory was affected. The function will calculate which pages of video memory are affected by this rectangle, put the “update required” flag on them and schedule the execution of the same callback that is called in case of deferred io. After that, all the operation callbacks will be reduced to calling the sys_ functions and the function we wrote, which we will call touch, similar to the command in Linux.
Display_touch function
static void display_touch(struct fb_info *info, int x, int y, int w, int h)
{
int firstPage;
int lastPage;
int i;
struct usblcd *dev=info->par;
firstPage=((y*WIDTH)+x)*BYTE_DEPTH/PAGE_SIZE-1;
lastPage=(((y+h)*WIDTH)+x+w)*BYTE_DEPTH/PAGE_SIZE+1;
if(firstPage<0)
firstPage=0;
if(lastPage>FP_PAGE_COUNT)
lastPage=FP_PAGE_COUNT;
for(i=firstPage;ivideopages[i].toUpdate);
schedule_delayed_work(&info->deferred_work, info->fbdefio->delay);
}
The code, I think, is understandable - we simply calculate which pages we touched on the basis of resolution, always round up, or rather, we simply take one page from each end with a margin.
Now we describe all the other callbacks - they become very simple:
All callbacks of operations
static void display_fillrect(struct fb_info *p, const struct fb_fillrect *rect)
{
sys_fillrect(p, rect);
display_touch(p, rect->dx, rect->dy, rect->width, rect->height);
}
static void display_imageblit(struct fb_info *p, const struct fb_image *image)
{
sys_imageblit(p, image);
display_touch(p, image->dx, image->dy, image->width, image->height);
}
static void display_copyarea(struct fb_info *p, const struct fb_copyarea *area)
{
sys_copyarea(p, area);
display_touch(p, area->dx, area->dy, area->width, area->height);
}
static ssize_t display_write(struct fb_info *p, const char __user *buf,
size_t count, loff_t *ppos)
{
int retval;
retval=fb_sys_write(p, buf, count, ppos);
display_touch(p, 0, 0, p->var.xres, p->var.yres);
return retval;
}
Now we will finally describe our callback from deferred io , which will send information to the display. It will largely coincide with the .write callback from the previous article. We will write in the display on the pages, not forgetting to attribute, as in the previous article, to them the required title. Thanks to our videopage structure , the x, y coordinates and lengths have already been calculated, so all you need to do is just push it into the buffer and drop it over USB.
Delayed I / O callback
unsigned char videobuffer[PAGE_SIZE+8];
static void display_update(struct fb_info *info, struct list_head *pagelist)
{
struct usblcd* dev = info->par;
int retval;
struct page *page;
int i;
int usbSent=0;
list_for_each_entry(page, pagelist, lru)
{
atomic_dec(&dev->videopages[page->index].toUpdate);
}
for (i=0; ivideopages[i].toUpdate))
atomic_dec(&dev->videopages[i].toUpdate);
else
{
*(unsigned short*)(videobuffer)=cpu_to_le16(dev->videopages[i].x);
*(unsigned short*)(videobuffer+2)=cpu_to_le16(dev->videopages[i].y);
*(unsigned long*)(videobuffer+4)=cpu_to_le32(dev->videopages[i].length>>1);
memcpy(videobuffer+8,dev->videopages[i].mem,dev->videopages[i].length);
retval = usb_bulk_msg(dev->udev,
usb_sndbulkpipe(dev->udev, 1),videobuffer,
dev->videopages[i].length+8,
&usbSent, HZ);
if (retval)
printk(KERN_INFO "usblcd: sending error!\n");
}
}
}
Since we can get into this callback in an honest way (via deferrd io ), or we can do it on our own initiative (by scheduling it to be performed in one of the callbacks of operations by calling display_touch), we simply go over all the pages passed to us, if any, and mark them as subject to update.
If we did not get here through deferred I / O, then the list will simply be empty.
After that, we simply go through all the pages atomically checking the need for an update and performing this very update by synchronous sending via USB. In the next article, when we finish the driver to its normal state, we will replace the synchronous sending with a more correct mechanism called the USB Request Block, or URB. It will allow the USB host to throw a request to send data and immediately return to further processing. And the fact that the URB flew (or did not fly) to the recipient will be reported to us in the interrupt. This will allow us to squeeze out a little more FPS from our system (without, however, exceeding the theoretical limit that I spoke about above).
We just had a little bit left - since we decided to do bad things and not clean ourselves, all that was left was to initialize everything in the Probe callback.
Callback Probe
static int LCDProbe(struct usb_interface *interface, const struct usb_device_id *id)
{
struct usblcd *dev;
struct usb_host_interface *iface_desc;
struct usb_endpoint_descriptor *endpoint;
unsigned char *videomemory;
int retval = -ENODEV;
int i;
dev_info(&interface->dev, "USB STM32-based LCD module connected");
dev = kzalloc(sizeof(*dev), GFP_KERNEL);
if (!dev)
{
dev_err(&interface->dev, "Can not allocate memory for device descriptor\n");
retval = -ENOMEM;
goto exit;
}
dev->udev=interface_to_usbdev(interface);
dev->interface = interface;
iface_desc = interface->cur_altsetting;
for (i = 0; i < iface_desc->desc.bNumEndpoints; ++i)
{
endpoint = &iface_desc->endpoint[i].desc;
if(usb_endpoint_is_bulk_out(endpoint))
{
dev->bulk_out_ep=endpoint;
dev->bulk_out_packet_size = le16_to_cpu(endpoint->wMaxPacketSize);
break;
}
}
if(!dev->bulk_out_ep)
{
dev_err(&interface->dev, "Can not find bulk-out endpoint!\n");
retval = -EIO;
goto error_dev;
}
dev->gdev = &dev->udev->dev;
dev->info = framebuffer_alloc(0, dev->gdev);
dev->info->par = dev;
dev->info->dev = dev->gdev;
if (!dev->info)
{
dev_err(&interface->dev, "Can not allocate memory for fb_info structure\n");
retval = -ENOMEM;
goto error_dev;
}
dev->info->fix = fixed_info;
dev->info->var = var_info;
dev->info->fix.smem_len=FP_PAGE_COUNT*PAGE_SIZE;
dev_info(&interface->dev, "Allocating framebuffer: %d bytes [%lu pages]\n",dev->info->fix.smem_len,FP_PAGE_COUNT);
videomemory=vmalloc(dev->info->fix.smem_len);
if (!videomemory)
{
dev_err(&interface->dev, "Can not allocate memory for framebuffer\n");
retval = -ENOMEM;
goto error_dev;
}
dev->info->fix.smem_start =(unsigned long)(videomemory);
dev->info->fbops = &display_fbops;
dev->info->flags = FBINFO_FLAG_DEFAULT|FBINFO_VIRTFB;
dev->info->screen_base = videomemory;
memset((void *)dev->info->fix.smem_start, 0, dev->info->fix.smem_len);
for(i=0;ivideopages[i].mem=(void *)(dev->info->fix.smem_start+PAGE_SIZE*i);
dev->videopages[i].length=PAGE_SIZE;
atomic_set(&dev->videopages[i].toUpdate,-1);
dev->videopages[i].y=(((unsigned long)(PAGE_SIZE*i)>>1)/WIDTH);
dev->videopages[i].x=((unsigned long)(PAGE_SIZE*i)>>1)-dev->videopages[i].y*WIDTH;
}
dev->videopages[FP_PAGE_COUNT-1].length=FB_SIZE-(FP_PAGE_COUNT-1)*PAGE_SIZE;
dev->info->pseudo_palette = &dev->pseudo_palette;
dev->info->fbdefio=&display_defio;
fb_deferred_io_init(dev->info);
dev_info(&interface->dev, "info.fix.smem_start=%lu\ninfo.fix.smem_len=%d\ninfo.screen_size=%lu\n",dev->info->fix.smem_start,dev->info->fix.smem_len,dev->info->screen_size);
usb_set_intfdata(interface, dev);
retval = register_framebuffer(dev->info);
if (retval < 0) {
dev_err(dev->gdev,"Unable to register_frambuffer\n");
goto error_buff;
}
return 0;
error_buff:
vfree(videomemory);
error_dev:
kfree(dev);
exit:
return retval;
}
Here we first allocate memory for our device descriptor, then for the framebuffer descriptor structure , and only then for the framebuffer itself. Please note - we select through vmalloc . The difference from kmalloc is that vmalloc can reconfigure the page tables of virtual memory, and “collect in pieces” the buffer we requested. That is, for us it will look like a single block of memory, but physically it can consist of pages that are not even close to each other. kmalloc returns memory, which is physically a single block. Since we are requesting a large enough piece, vmalloc is a good practice .
We compile, insmodem, and if everything is done correctly, we see the console on the display!
Conclusion
In this article, we finally made this important step - from the driver of a custom device that was not used by the system, we passed into the arms of the framework, which allowed us to tell all applications and drivers that we have a real display. Yes, we did a little bad, not cleaning up our resources and not handing off the device, so pulling out USB and plugging it into a working device is not recommended yet.
But we will definitely fix this in the next article. What do we have to do?
- Implement support for double buffering on the device, providing a two-fold increase in FPS.
- To fix our shortcoming, because of which the console is green (if we run anything other than the console, it will look awful. Maybe someone already guessed what was the matter?)
- Implement the correct cleaning of resources so that you can safely disconnect and connect the device
- Switch to using the asynchronous mechanism of URBs instead of sending the Bulk message synchronously, which will reduce losses in the driver
- Compile various fun apps to fully enjoy our mini-computer. So, for example, we will see what needs to be patched in the code of the Gobliins engine so that they finally start.
Below is another video with an old quest (no less beloved by me than Gobliins), a few photos of a small graphical shell that can be easily run on the device and the Elinks console browser.
The first Kirandiya
Habr in a console browser (font VGA8x8)
Shell Gmenu2X
Built-in file browser in it
Settings
I have everything for now. Successful implementation and until the next article!