PM tips and experiences : Measuring ranking systems
This is first in a series of notes (hopefully) where I pen down my thoughts on aspects of digital products . I am starting with how to measure ranking systems. This is by no means a technical paper and I want to avoid using technical terms as much as possible. I want to write this in a way that anyone with basic understanding of technology can follow. This is not the only way to measure ranking but is based on my learnings across companies and industries. (PS I used to run the hotel ranking team for Booking.com)
What is ranking and why is it important ?
Let me start by defining what is ranking and where you experience it in digital products. The Oxford English Dictionary says that ranking is “the action or process of giving a specified rank to someone or something”. Anytime you visit your Netflix profile you are seeing ranking in action. There are hundreds of thousands of shows available for you to watch on Netflix. Therein lies the problem. With so many to choose from how do you find the right one for you. Ranking comes to the rescue by ordering these shows in a particular order. Thus when you open your profile you feel like watching the first set of shows you see. Sometimes this order is based on your tastes and past history (personalised). Sometimes its not (Top 10 shows in your country). If Netflix messes up ranking you will not see the shows that are relevant to you. You thus end up being unhappy and quitting Netflix. This is good for Netflix competitors but definitely not good for Netflix. That is why companies invest huge amounts of money to make sure ranking works well. Do remember the customers can be internal users too.
Questions to ask ?
As a product manager managing a ranking system you will have to set goals for the team. Otherwise you do not know if you are making progress or not. How I like to think about goals is by asking some questions and then trying to measure them. As outlined before ranking is a way of ordering things. So you can ask 2 questions to measure if the way of ordering things is working.
Question 1. How well are the systems which are doing the ordering working ?
Question 2: How good is the ordering ?
Answer 1: Performance of ordering systems
The first question is about systems and engineering. You may have different goals based on where you are in the life cycle of the system. There will also be different metrics you can use to measure progress. You are the best judge to figure out which one is the best for you.
Answer 2: Performance of the ordering
This brings us back to the other important question. How good is the ordering ? To answer this lets define what good ordering means. Good can mean different things in different circumstances. To me the following come to mind. This list is not comprehensive by any means but just a thought starter.
So what makes an ordering good ? In my mind there are 3 ways .
1. Action oriented
Ordering that helps users find the item they are most likely to like / buy / click / book. For ease of reference I will call the item “most likely to action”.
What the action is will vary based on product / business. For a travel site it can be a flight booking. For a video streaming site it can watching a video. You can choose what is the action based on your understanding of the product. Sometimes you want people to find a product and seeing the details. Other times you care about them buying it, sometimes its about a customer adding it to the basket. You are in the best position to say what you want to drive. My only feedback there is be clear on what action you want to drive and why.
2. Ease oriented
Ordering that helps users find the “most likely to action” easily . This is different from the one above since the emphasis is on easy. Easy can be based on the order of the item on the list. So any system which helps users take action on the first item in the ranking is good. In the one above we are concerned about whether users are taking action or not.
3. Value oriented
Sometimes you care more about the total value driven and not the number of actions. Let me explain a bit in detail. For example, a hotel booking site may want users to book expensive rooms since it generates more commission.
This might be in conflict with an action oriented ordering. Even when customers are more willing to buy a $100 hotel room you may want to show them a $500 hotel room. Why ? Because you want to maximise revenue. Let’s say you have 10 customers and 2 ranking systems (A) & (B). When you use A you sell $100 hotels rooms to 7 customers. In system B you see only 2 $500 hotel rooms. In a value oriented ranking you may want to pick system B since you end up making more money (2 x $500 = $1,000 vs 7 x $100 = $700 ) per every 10 customers.
Value oriented ordering can get more and more complex depending on the situation. You may want to drive customer loyalty by focusing on Customer Lifetime Value. ( Sorry for the technical term). There are other times you may want to strike a balance between action and value oriented. For that you might need to come up with a complex metric you are optimising for. For example, a ranking score combining revenue earned (80% weightage) and orders generated (20% weightage).
In summary
Ranking systems help order items so that its easy for customers. You should think of how the system performs and how good the ordering is when you set goals. There are many ways you can measure goodness of ordering and there is no single right answer. Think of what you want to drive as a business and chose deliberately. If it’s wrong — change it, its not the end of the world. No product is perfect out of the door.
Hope you enjoyed reading this note. I had a good time writing it. Do let me know your feedback and improvement areas in the comments.
