Mean Average Precision
2016-07-04 10:20
330 查看
reference: https://www.kaggle.com/wiki/MeanAveragePrecision
Introduction
Parameters: nSuppose there are m missing outbound edges from a user in a social graph, and you can predict up to n other nodes that the user is likely to follow. Then, by adapting the definition of average precision in IR (http://en.wikipedia.org/wiki/Information_retrieval,http://sas.uwaterloo.ca/stats_navigation/techreports/04WorkingPapers/2004-09.pdf),
the average precision at n for this user is:
ap@n=∑k=1nP(k)/min(m,n)ap@n=∑k=1nP(k)/min(m,n)
where P(k) means the precision at cut-off k in the item list, i.e., the ratio of number of users followed, up to the position k, over the number k; P(k) equals 0 when the k-th item is not followed upon recommendation; m is the number
of relevant nodes; n is the number of predicted nodes. If the denominator is zero, P(k)/min(m,n) is set to zero.
(1) If the user follows recommended nodes #1 and #3 along with another node that wasn't recommend, then ap@10 = (1/1 + 2/3)/3 ≈ 0.56
(2) If the user follows recommended nodes #1 and #2 along with another node that wasn't recommend, then ap@10 = (1/1 + 2/2)/3 ≈ 0.67
(3) If the user follows recommended nodes #1 and #3 and has no other missing nodes, then ap@10 = (1/1 + 2/3)/2 ≈ 0.83
The mean average precision for N users at position n is the average of the average precision of each user, i.e.,
MAP@n=∑i=1Nap@ni/NMAP@n=∑i=1Nap@ni/N
Note this means that order matters. But it depends. Order matters only, if there is at least one incorrect prediction. The other words, if all predictions are correct, it doesn't matter in which order they are given.
Thus, if you recommend two nodes A & B in that order and a user follows node A and not node B, your MAP@2 score will be higher (better) than if you recommended B and then A. This makes sense - you want the most relevant results to show up first. Consider the
following examples:
(1) The user follows recommended nodes #1 and #2 and has no other missing nodes, then ap@2 = (1/1 + 1/1)/2 = 1.0
(2) The user follows recommended nodes #2 and #1 and has no other missing nodes, then ap@2 = (1/1 + 1/1)/2 = 1.0
(3) The user follows node #1 and it was recommended first along with another node that wasn't recommended, then ap@2 = (1/1 + 0)/2 = 0.5
(4) The user follows node #1 but it was recommended second along with another node that wasn't recommend, then ap@2 = (0 + 1/2)/2 = 0.25
So, it is better to submit more certain recommendations first. AP score reflects this.
Here's an easy intro to MAP: http://fastml.com/what-you-wanted-to-know-about-mean-average-precision/
Here's another intro to MAP from our forums.
Sample Implementations
our C# Production ImplementationR, test
cases
Haskell, test
cases
MATLAB / Octave, test
cases
Python, test
cases
Contests that used MAP@K
MAP@500: https://www.kaggle.com/c/msdchallenge/details/EvaluationMAP@200: https://www.kaggle.com/c/event-recommendation-engine-challenge
MAP@10: https://www.kaggle.com/c/FacebookRecruiting
MAP@10: https://www.kaggle.com/c/coupon-purchase-prediction/details/evaluation
MAP@5: https://www.kaggle.com/c/expedia-hotel-recommendations
MAP@3: https://www.kddcup2012.org/c/kddcup2012-track1/details/Evaluation
Article needs:
explanationformula
example solution & submission files
相关文章推荐
- VS 一些用法设置
- Verify the Developer App certificate for your account istrusted on your device.
- Android 常用Canvas
- 使用javascript原生XMLHttpRequest对象进行ajax交互
- 编写Javascript时的注意事项
- 文章索引-winform
- 读《挡不住的跨境电商时代》
- ugui基础控件button使用(二)
- OC学习总结之属性
- 获取的某个数据为null时,程序crash的解决办法
- Java基础——clone()方法浅析
- “Cannot load php5apache2_4.dll into server”问题的解决方法
- RecyclerView多种item类型头部底部
- NOIP 2012 借教室
- Spring3.2新注解@ControllerAdvice
- 文本相似度计算
- 有容云:DevOps 的起点-入手微型数据中心(上)
- SEO
- Android序列化:Serializable和Parcelable
- build\tools\buildinfo.sh