On many social news sites users can give and take points on comments. The purpose of the points is to give hints about what comments are worth reading. But upon closer inspection this method often fails to provide a way to let the sites software sort or highlight good comments.
A lot of time and effort is spent perfecting the algorithm used to generate the order of stories, but when it comes to user comments on each article the trend the past few years has been to allow any user to vote up or vote down a comment and to sort on the values. This up down method seems to provide very little real value and produces very poor results as a means of sorting.
When a user sees a comment the basic version of the algorithm that users go through is something like this:
"If I see a post that looks good I vote it up. If I think it is wrong I vote down."
The number of points a good post gets is directly related to the number of users that read the post. The more users read the post the more are likely to award it points. A good post A posted at time 0 could get 100 points and the same post made at time +1hour will get 50 points and the next day will only get 3 points. At the same time a bad post B posted at time 0 can still get 8 points.
Or to sum it up with one question:
Is a comment with 47 points better then one that has 46 points? Frequently you will find the answer to be yes, no, and maybe depending upon when the post was made and having very little to do with the actual content. Because comments are not created at the same time the points that they each get can't accurately be compared without taking into account extra information. You could take into account the time sense the article was posted, the number of people who saw the comment, the number of people who voted on comments and the total votes given at the time when the vote was cast. All this does it make it more complicated so the same system can be used rater then asking of the system should have been used in the first place.
For many of the social new sites the comments were second fiddle to the main page. Once the jewel of the site the article sorting was working then comments were added. Many social news sites the past few years have employed some sort of up/down voting system for articles. Developers being developers decided to reuse this same code on comments. Lets users up/down vote on any comment and sort based upon value and blam job done in no time flat. Of course the article sorting isn't actually based upon only the value of votes, but takes into account time as a factor of the ranking.
It seems as though sorting comments was created out of the laziness of developers. They do have one thing going for them though. When a site is small there will not be too many comments on articles so users can quickly read all of the comments. For a small number of comments the sorting doesn't matter that much. It is only when you get a large enough base to get many good comments that the issue comes to light. Of course by that time if you have lots of users there is no real incentive to make the site better because the incentive of making something better to get users is diminished.
While social news sites think that their secret sauce is the front page I wouldn't be surprised if the comments page was of much more value. After playing around with the issue for way to long I ended up with a model very similar to the one slashdot already has. For any given comment it can be ranked from -5 to +5 and given a type. You end up with comments that have ranks like: +5 funny. +4 informative, -3 troll, etc. It takes very few users to agree that something is informative or funny and should get a value of -3 or +5.
The number of points a good post gets is directly related to the value the users find the post to be. This will results in the comment changing values less and less as time goes by (reaching its true value). A good post A posted at time 0 will have 0 points, but two minutes later will be at 3 and from then on some might say it is 2 some might say it is 4, but it quickly approaches what everyone feels the comment actual worth is. Compared this to posting the comment A after one hour and it will probably again be marked at 3 by a few users. This allows for much better comparisons between comments.
Combined with the small point scale is adjectives. This makes the user not just say "I like this", but have to think about what they like which results in much better scores. It also gets the added benefit of allowing users to do things like automatically give -1 points for funny comments if they find them annoying (or +2 in the opposite case).
To add to the debate there are several live examples of the different systems (often times they even cover the same article!). First going back to the objective which is to rank or highlight good comments for readers so they don't have to read everything. This includes getting rid of junk/spam/trolls. Now head over to slashdot and click on an article from yesterday. View comments by default it shows +4 and higher. They are pretty good and informative. Now go to reddit. While it is able to get rid of spam comments it displays everything having no such thing as +4 or higher and even if you sort by Top you still find that that non-informative, but early comment (which may or may not be really good) is right at the top with a huge number of points and you have to scroll around reading everything, attempting to find a post of any real worth. The good comments are just as likely to be near the bottom with 4 points as near the top with 400. When a social news site is unable to bring good comments to the viewer a good chunk of the value of the social new site is lost.
Having points for comments is a system that is so easy to implement that it will probably never go away, but a closer inspection shows that it really doesn't do a good job of the basic feature it was created for which is to allow the system to give hints about what comments are worth reading.
While I doubt many sites will change what they are doing this will hopefully spawn some conversations for developers who are creating the next generation social new site. While the slashdot system might not be the best we need to take a step back and look to see what the objective of ranking comments and how to best achieve that.
Photo by
dougwwFor another fun comparison blog like this one see the blog post that discusses
two different ways to find the most attractive girls.
And for a blog discussing problems with giving users infinite karma points checkout Joshua Gross's
Qualifying “Web Karma”: It shouldn’t be a game.Update: Reddit has switched to a new comment system that attempts to resolve several of the problems listed. Read more on their blog:
reddit new comment sorting system
After more then a year of thinking about doing it I finally setup 
Posts
