❖ ❖ ❖
You need to do lots of split (aka A/B) testing and you may need to customize it
Therefore:
Choose testing tool(s) you can use right away and improve later.
- we used vanity out of the box for a long time.
- we use vanity for page level (e.g. homepage), feature level (e.g. presence or absence of “compare two store”) and within-a-page tests (e.g. presence or absence of list of produce on a family preferences page)
- Google has tools – as do other services, but they’re more focused on page-level testing. We wanted to be able to choose the granularity.
- We recently modified vanity to let us do splits other than 50%/50% as we wanted to use it to test with a smaller / controlled sample
- However, since vanity only measures one metrics per test, and since we often care about two or more metrics – we augment the reporting by combining vanity data with our event logs (HAVE METRICS ON EVERYTHING) to form our own views of the experiment results
❖ ❖ ❖
Steve Sanderson 