June 12, 2010 at 16:18Thoughts aloud about Protocol X
- Transfer
Two years ago, while working on Awesome , I joined the development of XCB , which is part of the Freedesktop initiative . I had to learn the secrets of the X11 protocol and the entire ancient and mysterious world surrounding it.
Over the past few months, I have finally washed off all this filth, and now I feel the need to share my thoughts on this mess, which has been going on for decades.
... Toto released their famous song "Africa" , and several smart guys worked on the window system - the X Window System (it's a full name), which also has a (too) long history. The final version of the protocol (11th) was developed in the 80s. You can find out the details of the story in a Wikipedia article .
In 2010, we are still listening to the disco, and still use the various protocols developed in the 80s (and even earlier than X). Music is evolving, protocols are changing, and the X11 is not standing still.
The problem is that the X11 does not evolve correctly. The guys from MIT and all sorts of other clever places created the first version of X in 1984 and made changes up to the 11th version of the protocol. It was released in 1987, and we use it to this day. Eleven versions were released in 3 years in full accordance with the “early release, frequent updates” model. I don’t know why, but this approach has ceased to be applied over the next 23 years (although the developers added (and removed) many protocol extensions).
I don’t know what changes were made in the first 11 major versions of protocol X, but I’m more than sure that over the past 20+ years there is a great need to make serious changes to it.
In my humble opinion, the X11 was not designed with the expectation that it will live 23 years. True, I do not blame anyone: I was a 4-year-old child and played Lego when the latest version of protocol X was released, so I had very little chance of doing anything better.
This is perhaps the main slogan of protocol X of recent years. Just don’t get me wrong: I'm not going to flog it.
The X11 protocol has grown old over the years, and developers have begun to add extensions to it . Over the years, tons of extensions have been added. In strict accordance with one of the original principles of X :
Without exception, all of them were added because (alas and ah) the X11 protocol did not provide for all things that appeared over the past 23 years, like video, OpenGL, multiple monitors, or the pleasure of drawing oval windows. Some of these extensions are still in use, and some have already sunk into oblivion.
Protocol extensions is not something bad, but trying to fix a protocol with extensions like XFixes seems like a bad idea to me, despite all the good intentions that Keith Packard could follow.
The X11 protocol (without extensions) defines about 120 types of requests: creating a window, moving a window, and so on.
Nowadays, at least 25% of them are completely useless: the use of fonts on the server side or drawing squares and polygons are not used by any modern application or toolkit. All of them are redefined in requests from extensions, such as, for example, XRender .
Working with multiple monitors is completely confusing. X11 was designed to work in Zaphod mode (independent monitors). But Xinerama , and nowadays XRandRcompletely supplanted it - the latest versions of X server (since about 2007) no longer support Zaphod mode, even though it is part of the core of the X11 protocol.
Even worse: many queries contain design limitations or flaws, such as those described in a paper by researchers from DEC: Why X is not our ideal window system .
Following its initial principles , X does not define policies, but only provides mechanisms that look right.
As a result, the developers got together and started writing specifications to define a set of rules and dogmas: ICCCM . That was 22 years ago, in 1988. It is useless to add that many points of this specification are already hopelessly outdated, because they did not take into account many modern technologies.
I am not the only one who thinks so. The authors of the projects, which became the two main desktop environments, KDE and GNOME , saw this already in the 90s, when I was just learning to count. And they wrote EWMH, another standard that is perched on top of ICCCM, adding to it a number of nice features (maximization, full-screen mode and others).
The problem is that this standard was also written by not very far-sighted people who simultaneously worked on their environments (GNOME, KDE, and maybe some more). These desktop environments then had and still have a number of powerful concepts that determine how the desktop should work: “it should have workspaces,” “a window can exist only on one workplace,” “we can only see one workplace at a time ”,“ we don’t have several screens, ”and so on.
The idea of how the desktop should work is now carved in marble for every application or library that implements EWMH, including GTK + and Qt .
As a result, now everyone just forgot about the standards. The toolkits implement them on their own, bypassing the limitations and flaws of the X11 protocol, and no one wants to look back.
And with these toolkits, people are treated as casually as they are with standards. This sometimes leads to unpleasant side effects. For example, Openoffice as a switch jobs .
A desktop with all of these poorly designed standards continues to evolve over a decade. New standards continue to be added to it, the most recent of which are based on D-Bus, for example, the Desktop Notification Specification or the latest Status Notifier Specification developed in KDE.
Status Notifier is a new implementation of the good old XEmbed- based system tray, but now using D-Bus mechanisms instead of X11, and adding the ability to show something more than just icons in the system tray.
The draft specification contains a serious design flaw that Wolfgang Draxfinger drew attention to in his appeal to the XDG mailing list. Wolfgang points out that X is a network-oriented protocol, but D-Bus is not. Therefore, using D-Bus with the Status Notifier specification to send system tray events is a bad idea: running a graphical application from computer A on computer B will update the tray on the wrong host!
From reading the correspondence, it’s obvious that this doesn’t scare some KDE developers :
What Oswald describes as a special case is for most of us the most typical situation. In general, someone as lucky.
In my opinion, this is a step back in the wrong direction. But we can also conclude that the network part of X has become useless, at least for KDE.
When I joined Freedestop, I had to work on the XCB library, XC Binding. XCB has a nice, clean, 21st century technology-based API for working with the X11 protocol. Its code is automatically generated based on the XML file with the protocol description.
For comparison, Xlib consists of a muddy code of 80 codes, almost uncommented and full of hard-coded things. Only a few are able to understand some of its nooks, such as internationalization or the implementation of XKB. And all of his code is synchronous .
I will explain for those who are not yet aware: X is a network protocol in which you must send a request (like GET in HTTP) in order to receive a response. Xlib forces the application to wait for a response to the request, so the application is blocked until the X server sends a response. XCB does not block the application, which allows you to send a series of requests, do something useful while waiting, and then get answers.
Imagine that your browser sends one request at a time to the web server and makes you wait for the page to display until it uploads the most recent image.
In the case when X and all its clients are located on the same computer, the delay is small and imperceptible, therefore, the benefit of XCB asynchrony is small. However, on a slow network, performance gains can be huge, whichproved by Peter Harris, rewriting xlsclients on XCB .
One of the long-term goals of XCB developers is to finally get rid of Xlib in order to increase speed and reduce the response time of X11 applications. To do this, you need to port many libraries, because almost none of them (with the exception of Cairo ) supports XCB.
From my point of view, now this is a rather useless work. The desktop world is entrusted to the GNOME and KDE projects, that is, GTK + and Qt. At the same time, it seems that not one of these toolkits is interested in working on either XCB or protocol X. They rather make great efforts to circumvent the existing limitations and flaws of X, and still sit on top of a mound of crutches and implementations of vicious- by design standards. It seems to me that no one wants to look back at all these levels and see how they can be improved.
They climbed too high to look down and see what kind of recoil might be.
Enlightenment with your EFLwas the first toolkit to have an XCB backend (thanks to the work of Vincent Torri). Unfortunately, this backend is no longer supported, and no one cares about it. The last time I took it, it didn’t even compile.
The Freedesktop wiki has a page called X12 that lists all the things that need to be fixed. Unfortunately, this list is only growing, and no one stutters about working on X12.
On the other hand, there are several people who work on XKB2 in their free time , the second version of the “ let's -try-fix-keyboard-part-protocol-written-23-years-ago” extension.
All in all, it doesn't look like the X12 will appear in the next decade.
Do we have alternative X? There is Wayland , but so far it’s pretty useless. There is also DirectFB , but it is poorly ported. In my opinion, there are no candidates for X replacement yet.
In any case, none of the main toolkits supports such an alternative. GTK + once supported DirectFB, but as far as I know, nothing is working now (as Josselin Mouette notes ). That is why the latest versions of the Debian installer have moved to use X in the graphical part (thanks to the work of Cyril Brulebois).
XCB has existed for more than five years, but very few have become interested in it. As far as I can see, no one is interested in using the X protocol, everyone just tries to encapsulate it with some high-level API to stop seeing it as soon as possible. This leads to the appearance of poorly written applications and toolkits full of ugly hacks.
All this also means that writing new applications and toolkits based on XCB should be a very interesting project, but it will take too much time to understand how to bypass the flaws of the X protocol introduced over the years by predecessors, including Qt and GTK + .
The main toolkits do practically nothing to win by returning to the dark waters of X. I think most of their developers prefer to work on pretty 3D effects based on geolocation than redefining the best foundation for everyone.
World X has too little labor. The lack of X maintainers on Debian is a simple consequence of this situation. Of course, there are very competent and skilled X developers, which you can easily see by reading blogs on Planet Freedesktop(I do not count). Unfortunately, their number is not enough to cover the entire scope of X: input devices, graphics devices, specifications for new protocol extensions and so on. Server X is a fairly late development, and most developers are more interested in working on it, rather than on the protocol itself. This can be understood.
I wonder where we will come with all this in a few years. By now, I have been boiling in a boiler called X for 3 years, and I feel that sooner or later all the alternatives to KDE and GNOME will die out. The time when you could choose between a dozen “modern” window managers has passed.
In the end, maybe it's a simple Darwinism as applied to computer software.
Translator's Note: The author of this article is French programmer Julien Danjou, a window manager developer for Awesome. The original article is located here .This is my lane. I have not practiced translations for a long time, so I will be grateful for corrections and clarifications.
Over the past few months, I have finally washed off all this filth, and now I feel the need to share my thoughts on this mess, which has been going on for decades.
When I was not yet ...
... Toto released their famous song "Africa" , and several smart guys worked on the window system - the X Window System (it's a full name), which also has a (too) long history. The final version of the protocol (11th) was developed in the 80s. You can find out the details of the story in a Wikipedia article .
In 2010, we are still listening to the disco, and still use the various protocols developed in the 80s (and even earlier than X). Music is evolving, protocols are changing, and the X11 is not standing still.
The problem is that the X11 does not evolve correctly. The guys from MIT and all sorts of other clever places created the first version of X in 1984 and made changes up to the 11th version of the protocol. It was released in 1987, and we use it to this day. Eleven versions were released in 3 years in full accordance with the “early release, frequent updates” model. I don’t know why, but this approach has ceased to be applied over the next 23 years (although the developers added (and removed) many protocol extensions).
I don’t know what changes were made in the first 11 major versions of protocol X, but I’m more than sure that over the past 20+ years there is a great need to make serious changes to it.
In my humble opinion, the X11 was not designed with the expectation that it will live 23 years. True, I do not blame anyone: I was a 4-year-old child and played Lego when the latest version of protocol X was released, so I had very little chance of doing anything better.
We will not correct. We will adapt
This is perhaps the main slogan of protocol X of recent years. Just don’t get me wrong: I'm not going to flog it.
The X11 protocol has grown old over the years, and developers have begun to add extensions to it . Over the years, tons of extensions have been added. In strict accordance with one of the original principles of X :
It is important to clearly define what the system is and what it is not. No need to try to serve the needs of the whole world; instead, the system should be extensible so that additional needs can be met in a suitable way.
Without exception, all of them were added because (alas and ah) the X11 protocol did not provide for all things that appeared over the past 23 years, like video, OpenGL, multiple monitors, or the pleasure of drawing oval windows. Some of these extensions are still in use, and some have already sunk into oblivion.
Protocol extensions is not something bad, but trying to fix a protocol with extensions like XFixes seems like a bad idea to me, despite all the good intentions that Keith Packard could follow.
It’s actually worse than you think
The X11 protocol (without extensions) defines about 120 types of requests: creating a window, moving a window, and so on.
Nowadays, at least 25% of them are completely useless: the use of fonts on the server side or drawing squares and polygons are not used by any modern application or toolkit. All of them are redefined in requests from extensions, such as, for example, XRender .
Working with multiple monitors is completely confusing. X11 was designed to work in Zaphod mode (independent monitors). But Xinerama , and nowadays XRandRcompletely supplanted it - the latest versions of X server (since about 2007) no longer support Zaphod mode, even though it is part of the core of the X11 protocol.
Even worse: many queries contain design limitations or flaws, such as those described in a paper by researchers from DEC: Why X is not our ideal window system .
We will pile on top even more broken standards
Following its initial principles , X does not define policies, but only provides mechanisms that look right.
As a result, the developers got together and started writing specifications to define a set of rules and dogmas: ICCCM . That was 22 years ago, in 1988. It is useless to add that many points of this specification are already hopelessly outdated, because they did not take into account many modern technologies.
I am not the only one who thinks so. The authors of the projects, which became the two main desktop environments, KDE and GNOME , saw this already in the 90s, when I was just learning to count. And they wrote EWMH, another standard that is perched on top of ICCCM, adding to it a number of nice features (maximization, full-screen mode and others).
The problem is that this standard was also written by not very far-sighted people who simultaneously worked on their environments (GNOME, KDE, and maybe some more). These desktop environments then had and still have a number of powerful concepts that determine how the desktop should work: “it should have workspaces,” “a window can exist only on one workplace,” “we can only see one workplace at a time ”,“ we don’t have several screens, ”and so on.
Dude, don’t worry: we have toolkits!
The idea of how the desktop should work is now carved in marble for every application or library that implements EWMH, including GTK + and Qt .
As a result, now everyone just forgot about the standards. The toolkits implement them on their own, bypassing the limitations and flaws of the X11 protocol, and no one wants to look back.
And with these toolkits, people are treated as casually as they are with standards. This sometimes leads to unpleasant side effects. For example, Openoffice as a switch jobs .
We do not want to look back? Worse, we forgot where we came from!
A desktop with all of these poorly designed standards continues to evolve over a decade. New standards continue to be added to it, the most recent of which are based on D-Bus, for example, the Desktop Notification Specification or the latest Status Notifier Specification developed in KDE.
Status Notifier is a new implementation of the good old XEmbed- based system tray, but now using D-Bus mechanisms instead of X11, and adding the ability to show something more than just icons in the system tray.
The draft specification contains a serious design flaw that Wolfgang Draxfinger drew attention to in his appeal to the XDG mailing list. Wolfgang points out that X is a network-oriented protocol, but D-Bus is not. Therefore, using D-Bus with the Status Notifier specification to send system tray events is a bad idea: running a graphical application from computer A on computer B will update the tray on the wrong host!
From reading the correspondence, it’s obvious that this doesn’t scare some KDE developers :
Of course, do not worry too much about this bizarre special case. At least you will consider it as such until you come across it yourself (just by testing something or setting up some sophisticated kiosk). "
What Oswald describes as a special case is for most of us the most typical situation. In general, someone as lucky.
In my opinion, this is a step back in the wrong direction. But we can also conclude that the network part of X has become useless, at least for KDE.
I prefer to believe in XCB
When I joined Freedestop, I had to work on the XCB library, XC Binding. XCB has a nice, clean, 21st century technology-based API for working with the X11 protocol. Its code is automatically generated based on the XML file with the protocol description.
For comparison, Xlib consists of a muddy code of 80 codes, almost uncommented and full of hard-coded things. Only a few are able to understand some of its nooks, such as internationalization or the implementation of XKB. And all of his code is synchronous .
I will explain for those who are not yet aware: X is a network protocol in which you must send a request (like GET in HTTP) in order to receive a response. Xlib forces the application to wait for a response to the request, so the application is blocked until the X server sends a response. XCB does not block the application, which allows you to send a series of requests, do something useful while waiting, and then get answers.
Imagine that your browser sends one request at a time to the web server and makes you wait for the page to display until it uploads the most recent image.
In the case when X and all its clients are located on the same computer, the delay is small and imperceptible, therefore, the benefit of XCB asynchrony is small. However, on a slow network, performance gains can be huge, whichproved by Peter Harris, rewriting xlsclients on XCB .
One of the long-term goals of XCB developers is to finally get rid of Xlib in order to increase speed and reduce the response time of X11 applications. To do this, you need to port many libraries, because almost none of them (with the exception of Cairo ) supports XCB.
From my point of view, now this is a rather useless work. The desktop world is entrusted to the GNOME and KDE projects, that is, GTK + and Qt. At the same time, it seems that not one of these toolkits is interested in working on either XCB or protocol X. They rather make great efforts to circumvent the existing limitations and flaws of X, and still sit on top of a mound of crutches and implementations of vicious- by design standards. It seems to me that no one wants to look back at all these levels and see how they can be improved.
They climbed too high to look down and see what kind of recoil might be.
Enlightenment with your EFLwas the first toolkit to have an XCB backend (thanks to the work of Vincent Torri). Unfortunately, this backend is no longer supported, and no one cares about it. The last time I took it, it didn’t even compile.
X12?
The Freedesktop wiki has a page called X12 that lists all the things that need to be fixed. Unfortunately, this list is only growing, and no one stutters about working on X12.
On the other hand, there are several people who work on XKB2 in their free time , the second version of the “ let's -try-fix-keyboard-part-protocol-written-23-years-ago” extension.
All in all, it doesn't look like the X12 will appear in the next decade.
Alternatives?
Do we have alternative X? There is Wayland , but so far it’s pretty useless. There is also DirectFB , but it is poorly ported. In my opinion, there are no candidates for X replacement yet.
In any case, none of the main toolkits supports such an alternative. GTK + once supported DirectFB, but as far as I know, nothing is working now (as Josselin Mouette notes ). That is why the latest versions of the Debian installer have moved to use X in the graphical part (thanks to the work of Cyril Brulebois).
Conclusion
XCB has existed for more than five years, but very few have become interested in it. As far as I can see, no one is interested in using the X protocol, everyone just tries to encapsulate it with some high-level API to stop seeing it as soon as possible. This leads to the appearance of poorly written applications and toolkits full of ugly hacks.
All this also means that writing new applications and toolkits based on XCB should be a very interesting project, but it will take too much time to understand how to bypass the flaws of the X protocol introduced over the years by predecessors, including Qt and GTK + .
The main toolkits do practically nothing to win by returning to the dark waters of X. I think most of their developers prefer to work on pretty 3D effects based on geolocation than redefining the best foundation for everyone.
World X has too little labor. The lack of X maintainers on Debian is a simple consequence of this situation. Of course, there are very competent and skilled X developers, which you can easily see by reading blogs on Planet Freedesktop(I do not count). Unfortunately, their number is not enough to cover the entire scope of X: input devices, graphics devices, specifications for new protocol extensions and so on. Server X is a fairly late development, and most developers are more interested in working on it, rather than on the protocol itself. This can be understood.
I wonder where we will come with all this in a few years. By now, I have been boiling in a boiler called X for 3 years, and I feel that sooner or later all the alternatives to KDE and GNOME will die out. The time when you could choose between a dozen “modern” window managers has passed.
In the end, maybe it's a simple Darwinism as applied to computer software.
Translator's Note: The author of this article is French programmer Julien Danjou, a window manager developer for Awesome. The original article is located here .