A Personalization A/B Test With a Lift for One Group and Nothing for Another
AWS's Machine Learning blog describes how Amazon Payments applied AI-based personalization to a product acquisition funnel, using a multi-objective contextual multi-armed bandit — a method that learns which option to show by trying candidates and tracking what happens. In a seven-week online A/B test, the company reports, one customer population saw a high single-digit percentage relative lift in final-funnel conversion while another saw no improvement over the existing experience. The figures are described in the present tense, so treat them as an interim read from a running test rather than a settled benchmark, and note that the lift and the flat result belong to different populations.
AI-generated discussion · Different perspectives
AWS's own account puts the blame for the flat segment on the creative work rather than the algorithm: the problem turned out to be the content, not the model.
Convenient diagnosis, coming from a party selling the model infrastructure. It may well be true, but it is also the least expensive explanation to offer.