Invitation to public discussion of CMS rating on usability

When choosing a CMS, a developer, as a rule, takes into account many of its parameters, such as cost, number of modules, popularity of the system, and, of course, the most important parameter is his knowledge of this CMS. However, at the same time, one very important characteristic drops out of consideration: how convenient it will be for the customer of the site to use it - add news, correct typos, etc.

This issue is especially acute when developing small corporate or private sites. In many small companies there are no dedicated specialists for working with the site, and often there is not even a full-time “computer specialist”. And the need to add or improve something on such a site turns into a big problem - the owner of the site "suffocates the toad" to pay "inadequate" money for small work, and the developer has no desire to "make free" tedious and uninteresting small things.

Periodically performing research on the interfaces of various CMS, I thought about what would probably be interesting and useful to do something like a convenience rating of different systems. And to orient this rating not to developers, but to site owners. He shared this idea with several web studios and found support and a noticeable interest in the topic. He turned to CMS magazine - and also met a rapport. Now the question is brought up for wide discussion by all interested parties.

To avoid suspicions of custom and jeans, discussion of all aspects of testing will be held as openly as possible, we are ready to consider any suggestions and arguments. The results of all tasks will be open, including videos of all sessions. Testing itself will be carried out using eye-tracking technology, which today is perhaps the most high-quality tool for researching the usability of sites and interfaces.

Of course, we understand that it is impossible to embrace the immensity. Some CMS are convenient for the store, while others are suitable only for the blog; some work well on small sites, but “lie down” under load or when a couple of hundred pages appear; some have gigantic functionality, but pay for it with complexity, etc. etc.

In order to somehow level these differences, we offer the following (all points are subject to discussion, comments and suggestions are welcome):

1. The rating will test the usability of CMS for a small corporate site: a dozen pages, company news, a couple of blogs. If the idea turns out to be interesting, then next time it will be possible to test, for example, shops ...
2. It explores the convenience of managing the site . Moreover, the customer will manage this site. Not having any experience with sites. Those. The situation is simulated when the studio made a website, handed it over to the customer and then he himself must do something with this website.
3. The tasks that will be solved during testing are recruited during the discussion - send your options. They should be:
   • Elementary. For example, “add news,” “correct a typo on a page,” “change the contact phone number,” “change the font size and color of the header,” “insert a picture on the page,” etc.
   • Have a clear performance indicator. The task is either completed or not.
   • Relate to reality, i.e. tasks should be of the same plan that confront the real owners of sites.
4. For each task, indicators such as the number of respondents who completed the task will be recorded; lead time; number of errors during execution; as well as data on the movement of the gaze of respondents.
5. The team of each CMS makes a website on it and “sharpenes” it under the tasks that will be solved during testing. But it “hones” with regular means: if a quick access panel is provided in the CMS, then you can place a call to the corresponding modules there; but you can’t write custom scripts that display the “change title” button on the site.
6. We invite all CMS developers, as well as a support team for free CMS to participate. Studios are also invited who are ready to play for some kind of CMS (in the event that the developer refuses to exhibit his team). Send comments about what CMS I would like to see in this rating.
7. As respondents, people who know about the Internet and have seen a computer, but have no experience writing sites, will be recruited.
8. Financial questions. Unfortunately, conducting such a large-scale study is a rather expensive thing. So sponsors and advertisers are wanted. If you are interested in participating, write to eyetracker.ru@gmail.com and cmsmagazine@gmail.com

We hope that the results of such a study will be interesting not only as a rating, but also as an assessment of the "usability" of solutions used in the interfaces of various CMS. In parallel with the rating, usability research reports will be prepared for each participating CMS, which may be of interest to both their developers and other interface developers.

PS. Last year, there was already some attempt to objectively compare the two popular CMS (albeit in other ways), which led to a noticeable holivar in the discussion. This time I want to avoid wars, so all constructive proposals and ideas are accepted.