Evaluating Recommenders
Recommender evaluation is unusually treacherous: the offline metric is computed on data generated by a previous model, the ranking metrics disagree with each other, and the correlation between offline improvement and online business impact is famously weak.
Online Evaluation and A/B Testing
The offline metric improved, and the business metric did not. This gap is common enough to expect it as the default outcome, not a surprising exception — offline metrics are proxies, and only a genuine online experiment measures the thing actually being optimised for.