Rating Mechanisms. Sad experiences and optimistic conclusions.

    Humanity has always followed the path of social inequality. Someone was better, someone worse, according to certain criteria. Someone in power was chosen, appointed, or seized this power, organizing revolutions and building new societies. All this is successfully transferred to the Web. Websites are created, administrators appoint moderators, and users vote, measured by "karma". For them, “karma” (rating, popularity - as you like) is a sign of superiority.

    I want to talk about the experience that I received as the creator of one regional portal for creative people.

    Two years ago, thinking about user rating mechanisms, I came up with the following scheme:
    - initially the user has a rating of 0;
    - A user with a zero rating can put a rating of 10 points to a job they like;
    - the rating of works is summed up and the user rating is calculated;
    - upon reaching a certain rating, the user can give work ratings of 10 and up to 50 points.

    What did I try to achieve in this way? To weed out inactive users who register but do not post work, they will never earn a high enough rating in order to somehow influence the rating of work and avoid markups. I naively believed that any work would receive a rating from 10 to 50 from each user. In practice, it turned out that the user in most cases voted to the maximum or did not vote at all. In addition, personal sympathies unrealistically boosted the rating, as a result, someone was on the horse and someone was an outsider, both of which were good in their work.

    I had to change the voting system.

    Since the site was creative, it was decided to make several scales for evaluating the work: creativity (originality), artistry, technology. On each of the scales, it was possible to put a rating from 0 to 100. The following principles were:
    - each rating had a weight;
    - the more works the user has and the more the reting of these works, the greater the weight of the user's ratings;
    - weight of user ratings with a zero rating - zero.
    This system proved to be better and at first it seemed a panacea. But the following situation arose: users began to perceive ratings below 50 on any scale as an unreasonable underestimation and even had to introduce an item in the site’s rules that obliged to comment on their low ratings (I will make a reservation for clarity - the author of the work knew who and what rating he put, maybe this became the reason for this situation). In addition, many complained that it was very difficult to choose how much to put: 88 or 92, too much a run. In addition - there were works that had a zero rating but were not bad - they simply did not vote for them, although other works were worse than them but had a rating, albeit a low one.

    Now, for subjective reasons, it is not possible to introduce a new voting system, but a new site is being prepared for release (which I hope to present on the Habré (I hope on the Superhabr :)) and there I want to take into account the mistakes of the past:
    - do not let users see who and how evaluates them;
    - do not make the user think;
    - “If there is paradise, then there is hell”, give the opportunity to minus the work.

    One question remains, is the -1 / + 1 system so good?

    Also popular now: