Something's been bothering me for a while now. VinCellar's trust ratings don't add up and sometimes are just plain wrong. A couple of weeks ago I took a look at how we calculate our ratings and noticed that the calculations did not really capture the nuances of tasting notes and voting.
A simple summary of the old algorithm is that it only considered reviews with votes
when deriving a rating for an author. How is this flawed? I'll use an example to illustrate. A user with 20 reviews and one vote is judged solely on that vote. If the vote is positive he's at 100%, if it's negative he's at 0%. What about the other 19 notes? These should count for something.
How our new rating system works
In an attempt to protect authors who have numerous notes (and even those with a few) we've altered the rating system to allow unrated notes to contribute a score to the overall rating. I like to call this the "Innocent until proven guilty" trust rating system. Also all votes on a single review only contribute a score to that review. The algorithm is optimistic in that the score used for unrated notes is the same as if the note was positively rated. Here's a technical view of it:
So, using the same example our user with 20 reviews (1 rated, 19 unrated) would now have the following trust rating:
and using the example of a user with only two reviews, one unrated and one rated, his trust rating would be as follows:
Not too bad. Here's the key point of the algorithm, multiple votes on a single review now only affect that review's contribution to the author's overall trust rating and an author gets the benefit of the doubt on unrated reviews until someone votes on them.
What this means to you
- Trust ratings are fairer - In general, everyone's rating will be higher. All users, prolific or not, will get the same treatment so you can add your first notes without worrying about having a low rating.
- Write more tasting notes- The more reviews you write, the less likely your trust rating is dented by negative reviews.
- Vote more - Votes only count on their corresponding review. Give someone the thumbs up when you like what they have to say, you also won't damage their rating too much if you vote a review down.
- Understand the system - We know that this algorithm could be more sophisticated and we're open to refinements.
We'd love to hear what you think; you can follow the discussion in the forums. Thanks for your time.
