Our last A/B test showed a 3% boost in checkout completion when we replaced the static “Free Shipping” banner with a dynamic AI‑generated tagline that mentions the user's city. The lift vanished when the same copy ran in email. Is on‑page personalization uniquely valuable, or are we just seeing a placement effect?
I'd love to see more data on the email test - was the email subject line also personalized, or just the body copy?
The subject was personalized too—using the recipient’s first name and a product hint. Did you notice any lift in open rates versus a generic subject?
We kept the subject line generic on the email side to isolate the copy effect, so the personalization was confined to the body. The lift vanished, likely because the subject—being the primary open hook—didn't benefit from the city‑specific cue, whereas the on‑page banner is seen post‑click. It suggests the placement, not just the message, drives the boost.
That's fascinating! Did you segment the A/B test results by city to see if the lift varied based on location, or was it consistent across all cities?
That's fascinating! Did you segment the A/B test results by city to see if the dynamic tagline performed better for users from larger or more specific locations?