March 28, 2007 at 08:32

Rating of web studios "TOP-100" - review of criticism and comments of the organizers

    I will make a reservation right away - I wrote this material specifically for Habr. Habrahabr is a community on which the most adequate discussions are held (and criticism is mainly reasoned), plus there is a hit in the target audience.
    After the start of the project “TOP-100 Rating of the leading Runet web studios”, a lot of discussion, suggestions and criticism arose on various resources. Part of the criticism can be considered justified, and therefore I would like to clarify our position why we are implementing the project in this way, telling, in fact, how we are implementing it, and also answer a number of questions that were addressed to us, including Habré.
    First, I will talk very briefly about the project. Whoever has not seen, can get on the site and get detailed information.
    Briefly about the project
    The project is a study of the website development market in Runet. As part of the project, a survey of representatives of web studios and independent experts is conducted to determine the quantitative indicators of the market (volume, number of players, distribution by region, portrait of an average studio, etc.), as well as build the TOP-100 leading companies in terms of market share ( according to market participants and experts). The project website is http://2007.tagline.ru .
    The project started on March 14th. During the pre-registration week, more than 300 web studios registered in the project framework and a number of independent experts were involved, as well as information sponsors. Now - from March 23 to 30, there is a vote. Companies that do not have time to register previously have the opportunity to do this right now during the survey. The results of the study and the rating itself will be published on April 3.
    We now turn to the main questions and claims that are presented by critics:
    The main complaints of critics
    If we take justified criticism, we can distinguish several basic questions and claims addressed to the project. Let's try to answer them and explain our position:
    • The rating is biased, because it is not based on real data on studio turnovers, but only on the opinions of representatives of studios and experts, whose awareness is another question.
      From the point of view of a real assessment of company turnover, the rating really cannot be called 100% reliable. However, we are not talking about this anywhere. On the contrary, wherever possible, we emphasize that the rating reflects only the opinion of the professional environment on the distribution of leaders, and not only the turnover of the company, but also the level of its fame play an important role in experts' assessments. However, with this reservation, the information provided by us is reliable and can be used as a real tool. The problem is that at the moment it is impossible to create an objective rating precisely by turnover - the market is too smeared, there are a huge number of small players whose turnover is simply impossible to calculate. Large companies also do not disclose their momentum. Making estimates on the portfolio or the number of employees adjusted for the rate of return - you can be wrong at times. And if you make a mistake with one company as a plus, and with another as a minus, you get a difference in the estimation of turnover by an order of magnitude for equal companies. Therefore, it is now impossible to form a rating based on objective and reliable estimates. As soon as, and we hope so, the market situation will cease to be so opaque, we will immediately provide the public with data on the actual distribution of turnover.
    • It is not clear why this rating is needed at all? It turns out the inter-studio between the studios, but no one will really disclose interesting data.
      To the question why rating and research are needed, I will answer below. Further, the current scale of the project (more than 300 studios) does not allow talking about a cabal - the number of respondents is too large. As for the interestingness of the data - will it really be interesting for people working in this market to hear the opinions of dozens of leaders, experts and just a large number of their colleagues who do not need to distort information on general market issues? I will not believe.
    • Rating - PR campaign of someone there, all places are bought.
      This is non-constructive criticism, although quite frequent. I will not answer it, because it’s funny. And it will be ridiculous to any person who knows the market and sees the composition of experts.
    • The research data is not needed by anyone and will show only the "average temperature in the hospital."
      It depends on how you teach them. We do not plan to stupidly average all the data obtained in the survey. We will provide a distribution of all parameters, as well as conduct a thorough analysis of all interesting indicators. What exactly, what reports and parameters will be included in our study - I wrote below.

    By the way, in terms of a constructive analysis of the rating methodology last year, interesting material was written by Dmitry Vasiliev, director of the development company Aist.

    Why and who needs rating and research?

    I would also like to dwell on the question of why and to whom the rating and market research can be interesting and useful.

    The target audience:
    • Developers, representatives of studios The
      rating is interesting to representatives of studios in many positions. Firstly, the self-identification of its place in the market. Secondly, sports interest in whether we hit the rating. Thirdly, obtaining real information about the state of the market. And fourthly, the commercial effect of potential customers who will use the rating.
      The study is interesting to developers even more so. For the competent work of any company, she needs information about the market in which she works. The market for creating sites is in an information vacuum. One fact that before me last year no one estimated the size of the market - already says a lot. Most companies do not have marketing activities on the market, and information is very necessary. Actually, the idea to carry out such a project came to my mind when real data on the market were really needed - but they were not there.
    • Customers
      If customer research data is mainly needed as supporting information to broaden one's horizons, then the rating can be turned into a real tool. I do not urge in any case the customer to use the rating as the main criterion for selecting a contractor, this would be stupid. But as an auxiliary tool, one of the filters for selecting a contractor is why not. Moreover, following the results of last year’s rating, I regularly hear about various tenders and projects (both for national and global brands, as well as for local companies), where the circle of contractors was compiled taking into account the rating. Again, given the lack of analytical information on the market, the customer needs to be guided at least by something, and the rating (with a correct understanding of the method of its formation) can be a significant help. It must be remembered

      Ultimately, a careful reading of the research results and ratings will help customer representatives avoid many of the mistakes they tend to make. Also, as, for example, an article I wrote for Habr about typical managerial errors made by a customer when developing a site.
    • Interested / newcomers The
      project gives them interesting information for thought, which will help them if they ever come across the market in practice.

    What will be included in the study and how it is carried out
    Let us tell you what kind of information and applications we plan to include in the final report and rating:

    General market research
    • Analysis of the current market situation (+ selected expert comments)
    • Identification of the main market problems (+ selected expert comments)
    • Quantitative data on the market and their analysis: (+ selected expert comments)
    • Main market development trends (+ selected expert comments)

    TOP-100 Studios
    • Final rating TOP-100 of leading Runet Studios 2007
    • Detailed ranking analysis, comparison with last year, identification of trends
    • Selected expert comments on their voting options
    • Comments "a posteriori" of the representatives of the studios from the TOP-10
    • “Small rating” of studios according to independent experts, comparison

    Applications
    • All comments of experts and studio representatives on all general issues.
    • Distribution of studios by site development platforms
    • Distribution of studios by type of use of their / third-party CMS-systems

    The rating is based only on the results of a survey of studio representatives and experts. The study is based both on the results of the survey and our group’s own data. As part of our own research, a catalog of more than 800 existing studios was compiled, and various information on them was collected and analyzed. sample evaluations of various market segments were also carried out, data from a large number of customer companies on the distribution of Internet budgets were collected, parallels were made with the Internet advertising market, and a number of other research problems were also solved. Based on these data, final reports will be generated.
    In conclusion, I would like to add that our project is always open for constructive criticism, as well as any ideas. We plan to conduct a number of market research using other methodologies that we will try to conduct in the near future.
    Related materials:
    And finally - a brief overview of the materials on the project and its discussions, so that a complete picture is formed:



    Tags:

    Also popular now: