July 26, 2007 at 05:05Ranking Criteria: From each according to ability, to each according to work (No. 3)
The purely socialist slogan put in the title of this note, the third in the series “How do we equip the social web?”, May turn out to be not bad at all with respect to Web 2.0. Yes, and if you think about it well, it’s not so bad for real life. The only trouble is that no one has ever intended to truly use it in this life.
In the previous noteof this series (all notes of the series will be marked with their relative serial number) we examined the tools with which each registered user can express their attitude to various input messages coming from other users. In this, we will dwell on why, how, when, and to what extent, when ranking information and users, the system (resource, ISR) should use the assigned estimates.
In the proposed approach to the selection of the main criterion for ranking participants, direct user ratings are either not taken into account at all or they come with less weight, indirect actions of readers that objectively reflect their real interest in the messages of the evaluated participant. Indeed, it is really difficult to assume that you will get acquainted with materials that are uninteresting and useless to you, vote on them and participate in their discussions. In other words, practice (your actions, not intentions and addictions) is a criterion of truth.
What is it about ?
In general, objects of ranking in social resources are subject to three types of objects :
1. Main messages (topics / topics)
2. Users
3. Sites(resources - sources of information)
The following may serve as initial information for ranking:
1. Direct user ratings :
- of an object , for example, as we saw in the previous note with respect to messages
, of another object , for example, a message when rating its author
2. Indirect actions of users who can say something about interest in a ranked object
And now we will consider all this more specifically in relation to each of the three types of ranking objects. As always, we will first talk about existing resources, and then about what is being offered for a typical intelligent social resource (ISR). At the same time, I want to immediately warn that, speaking of existing resources and in some cases not knowing the exact algorithms for their work, I will rely on my observations. The opacity of ranking algorithms is a matter of a separate conversation.
Ranking of main messages The
rank of a message in existing resources is defined as:
digg - a function of the sums of all user pros and cons ( Bury)) in relation to this message. Such an integral rating is here called Message Popularity.
reddit - some function of the sums of all the user’s pros ( Up / like ) and cons ( Down / dislike ) in relation to this message. However, such a function is considered here only as one of the ranking options. Almost equally, others are used, for example, by Controversy and Hot (
news ) relevance2 - a certain function of the sums of all user’s pros ( Mark ) and cons ( problem? ) In relation to this message. Such an integral estimate is here called Quality of Communication..
habrahabr - some function of the sums of all user pros and cons in relation to this message. Such an integral estimate here is called Habrasila
del.icio.us - a certain function of the sums of all user “ save it ” in relation to this message. Such an integral estimate is here called the message popularity .
bobrdobr.ru - message ranking is not used (or did it seem to me?)
I have a suspicion that the values of the arguments for and against in some resources are also affected by the ranks of voting users. I also think that sometimes the amount of comments on this post also affects the value of this weighting factor. With a higher probability, the latter assumption is true for reddit when ranking by “inconsistency” and relevance.
It is proposed for the ISR:
1. Ranking by nominations - multi-criteria ranking
2. Equality of all nominations - the user chooses which nomination suits him best and can even correlate such a nomination with his homepage
3. Relevance of the messageIt is the most “objective” nomination for messages, although it is considered by the system only as the first among equals. It is precisely by the criterion of relevance that the messages displayed on the main page by default are ranked. As we will see later, the same criterion is the main (if not the only) used in ranking users.
4. Equality of "rating" users in terms of the ability to express their rating of any message (not yours, of course). As for the weighting coefficients of such an assessment, the status of the voter in certain cases may affect such a weighting factor. But it is planned to devote a separate conversation to this.
The formulated approach, from my point of view, is a guarantee of a high level of ISR pluralism and prevents the ability of some group of users to ever impose their “ideological” preferences on others.
First, a few words about the ranking nominations of the main messages, each of which is associated with its own criterion. Generally speaking, there can be as many as you like. I will give only a few (names are conditional; like the nomenclature of nominations, this is a matter of taste for the developers of the IRS):
- The relevance (popularity, demand) of messages - as we have already noted, this is the main criterion. Therefore, from my point of view, it is mandatory for any IRS. See below for more details on this nomination.
- Recognition of messages- contain ideas (thoughts, suggestions, conclusions) with which most who have familiarized themselves with them
- Inadmissibility of messages - contain ideas (thoughts, proposals, conclusions) with which most who have familiarized with them do not agree
- Messages are inconsistent - contain controversial ideas (thoughts, suggestions, conclusions).
Consider in more detail the ranking criterion for the relevance of the information. Other ranking criteria (nominations) will be presented in the next note.
The rank of the message in the nomination of relevance is defined as a function of the sums of all user:
1. views (View)this post. Of course, it means that the user first receives only a general list of messages.
2. direct answers - top-level comments to this post
3. votes I agree - I disagree , no matter how each user specifically voted
4. comments from lower levels related to this post
5. votes to these comments , again independently from what they are
Each argument (sum) has its own weight coefficient, the choice of which can play a decisive role in the correctness of the nomination. Without delving into the subtleties of this issue, we can, in general, outline the following priorities (in decreasing order of the weight coefficients): 2, 1, 3, 4, 5. Moreover, for arguments 4 and 5, related to the lower level comments, the weight coefficient should drop sharply for each next level.
Why do I still prefer comments in relation to the usual views (reads) of messages. The answer seems simple to me. We are all masters of making catchy annotations , as I did on this note on the advice of a respected sheller. At the same time, it’s hard to disagree with dik’s statement that behind a “tempting preview”, one often “hides a frank dummy”(see comments on the previous post in habrahabr). I hope this does not apply to my notes :). But each of us, as a rule, will write a comment only if his material has touched the living.
As we can see, here direct user ratings are either not taken into account at all (for example, “like - dislike”) or go with less weight (“agree, disagree”) than his indirect actions, which objectively reflect the user 's real interest in this post. Indeed, it is really difficult to assume that you will get acquainted with materials that are uninteresting and useless to you, vote on them and participate in their discussions. In other words,practice (your actions, not intentions and addictions) is a criterion of truth .
Just one example to illustrate what has been said. I have one faithful reader - my sworn friend. No, not the one that you thought about - everything is within the permissible limits, although sometimes it pulls on Something wrong . Who is he and where from, now does not matter. The main thing is that “my friend” does not miss almost any of my notes. At the same time he cleans me as soon as he can and sets up the corresponding minuses. Almost a rhetorical question, are my thoughts relevant to him or not? Or maybe it's just such a modern informational Robin Hood, who has devoted his whole life to protecting orphaned and poor users of the social web?
It is also easy to notice that in defining the criterion of relevance I follow the well-known rule: “the new is the well-forgotten old”. After all, remember what postings we primarily read on forums built on traditional engines like IPB? That's right, those that have been viewed and commented on most by others before us. True, in the forum engines known to me in the output listings, messages are rarely sorted by the criterion of relevance, but the times are different now :). This, by the way, is once again to the hint that sounded to me about the use of the “converted forum”.
And now it will be easier for us to move on to the very “sick issue of the social sphere”. To the question of who, how and why you and I are ranking. Read about it in the extended version of the note., located in iTech Bridge (Attention! I honestly warn you, the note turned out to be quite long again) .
In the previous noteof this series (all notes of the series will be marked with their relative serial number) we examined the tools with which each registered user can express their attitude to various input messages coming from other users. In this, we will dwell on why, how, when, and to what extent, when ranking information and users, the system (resource, ISR) should use the assigned estimates.
In the proposed approach to the selection of the main criterion for ranking participants, direct user ratings are either not taken into account at all or they come with less weight, indirect actions of readers that objectively reflect their real interest in the messages of the evaluated participant. Indeed, it is really difficult to assume that you will get acquainted with materials that are uninteresting and useless to you, vote on them and participate in their discussions. In other words, practice (your actions, not intentions and addictions) is a criterion of truth.
What is it about ?
In general, objects of ranking in social resources are subject to three types of objects :
1. Main messages (topics / topics)
2. Users
3. Sites(resources - sources of information)
The following may serve as initial information for ranking:
1. Direct user ratings :
- of an object , for example, as we saw in the previous note with respect to messages
, of another object , for example, a message when rating its author
2. Indirect actions of users who can say something about interest in a ranked object
And now we will consider all this more specifically in relation to each of the three types of ranking objects. As always, we will first talk about existing resources, and then about what is being offered for a typical intelligent social resource (ISR). At the same time, I want to immediately warn that, speaking of existing resources and in some cases not knowing the exact algorithms for their work, I will rely on my observations. The opacity of ranking algorithms is a matter of a separate conversation.
Ranking of main messages The
rank of a message in existing resources is defined as:
digg - a function of the sums of all user pros and cons ( Bury)) in relation to this message. Such an integral rating is here called Message Popularity.
reddit - some function of the sums of all the user’s pros ( Up / like ) and cons ( Down / dislike ) in relation to this message. However, such a function is considered here only as one of the ranking options. Almost equally, others are used, for example, by Controversy and Hot (
news ) relevance2 - a certain function of the sums of all user’s pros ( Mark ) and cons ( problem? ) In relation to this message. Such an integral estimate is here called Quality of Communication..
habrahabr - some function of the sums of all user pros and cons in relation to this message. Such an integral estimate here is called Habrasila
del.icio.us - a certain function of the sums of all user “ save it ” in relation to this message. Such an integral estimate is here called the message popularity .
bobrdobr.ru - message ranking is not used (or did it seem to me?)
I have a suspicion that the values of the arguments for and against in some resources are also affected by the ranks of voting users. I also think that sometimes the amount of comments on this post also affects the value of this weighting factor. With a higher probability, the latter assumption is true for reddit when ranking by “inconsistency” and relevance.
It is proposed for the ISR:
1. Ranking by nominations - multi-criteria ranking
2. Equality of all nominations - the user chooses which nomination suits him best and can even correlate such a nomination with his homepage
3. Relevance of the messageIt is the most “objective” nomination for messages, although it is considered by the system only as the first among equals. It is precisely by the criterion of relevance that the messages displayed on the main page by default are ranked. As we will see later, the same criterion is the main (if not the only) used in ranking users.
4. Equality of "rating" users in terms of the ability to express their rating of any message (not yours, of course). As for the weighting coefficients of such an assessment, the status of the voter in certain cases may affect such a weighting factor. But it is planned to devote a separate conversation to this.
The formulated approach, from my point of view, is a guarantee of a high level of ISR pluralism and prevents the ability of some group of users to ever impose their “ideological” preferences on others.
First, a few words about the ranking nominations of the main messages, each of which is associated with its own criterion. Generally speaking, there can be as many as you like. I will give only a few (names are conditional; like the nomenclature of nominations, this is a matter of taste for the developers of the IRS):
- The relevance (popularity, demand) of messages - as we have already noted, this is the main criterion. Therefore, from my point of view, it is mandatory for any IRS. See below for more details on this nomination.
- Recognition of messages- contain ideas (thoughts, suggestions, conclusions) with which most who have familiarized themselves with them
- Inadmissibility of messages - contain ideas (thoughts, proposals, conclusions) with which most who have familiarized with them do not agree
- Messages are inconsistent - contain controversial ideas (thoughts, suggestions, conclusions).
Consider in more detail the ranking criterion for the relevance of the information. Other ranking criteria (nominations) will be presented in the next note.
The rank of the message in the nomination of relevance is defined as a function of the sums of all user:
1. views (View)this post. Of course, it means that the user first receives only a general list of messages.
2. direct answers - top-level comments to this post
3. votes I agree - I disagree , no matter how each user specifically voted
4. comments from lower levels related to this post
5. votes to these comments , again independently from what they are
Each argument (sum) has its own weight coefficient, the choice of which can play a decisive role in the correctness of the nomination. Without delving into the subtleties of this issue, we can, in general, outline the following priorities (in decreasing order of the weight coefficients): 2, 1, 3, 4, 5. Moreover, for arguments 4 and 5, related to the lower level comments, the weight coefficient should drop sharply for each next level.
Why do I still prefer comments in relation to the usual views (reads) of messages. The answer seems simple to me. We are all masters of making catchy annotations , as I did on this note on the advice of a respected sheller. At the same time, it’s hard to disagree with dik’s statement that behind a “tempting preview”, one often “hides a frank dummy”(see comments on the previous post in habrahabr). I hope this does not apply to my notes :). But each of us, as a rule, will write a comment only if his material has touched the living.
As we can see, here direct user ratings are either not taken into account at all (for example, “like - dislike”) or go with less weight (“agree, disagree”) than his indirect actions, which objectively reflect the user 's real interest in this post. Indeed, it is really difficult to assume that you will get acquainted with materials that are uninteresting and useless to you, vote on them and participate in their discussions. In other words,practice (your actions, not intentions and addictions) is a criterion of truth .
Just one example to illustrate what has been said. I have one faithful reader - my sworn friend. No, not the one that you thought about - everything is within the permissible limits, although sometimes it pulls on Something wrong . Who is he and where from, now does not matter. The main thing is that “my friend” does not miss almost any of my notes. At the same time he cleans me as soon as he can and sets up the corresponding minuses. Almost a rhetorical question, are my thoughts relevant to him or not? Or maybe it's just such a modern informational Robin Hood, who has devoted his whole life to protecting orphaned and poor users of the social web?
It is also easy to notice that in defining the criterion of relevance I follow the well-known rule: “the new is the well-forgotten old”. After all, remember what postings we primarily read on forums built on traditional engines like IPB? That's right, those that have been viewed and commented on most by others before us. True, in the forum engines known to me in the output listings, messages are rarely sorted by the criterion of relevance, but the times are different now :). This, by the way, is once again to the hint that sounded to me about the use of the “converted forum”.
And now it will be easier for us to move on to the very “sick issue of the social sphere”. To the question of who, how and why you and I are ranking. Read about it in the extended version of the note., located in iTech Bridge (Attention! I honestly warn you, the note turned out to be quite long again) .