Evaluating Recommenders
Recommender evaluation is unusually treacherous: the offline metric is computed on data generated by a previous model, the ranking metrics disagree with each other, and the correlation between offline improvement and online business impact is famously weak.
The Recommendation Problem
Recommendation looks like supervised learning and behaves nothing like it. The data is almost entirely missing, the missingness is not random, the output is a ranked list rather than a prediction, and the model's own outputs determine what data you collect next.