Have A Testing Solution

❖ ❖ ❖

You need to do lots of split (aka A/B) testing and you may need to customize it


Choose testing tool(s) you can use right away and improve later.

  • we used vanity out of the box for a long time.
  • we use vanity for page level (e.g. homepage), feature level (e.g. presence or absence of “compare two store”) and within-a-page tests (e.g. presence or absence of list of produce on a family preferences page)
  • Google has tools – as do other services, but they’re more focused on page-level testing. We wanted to be able to choose the granularity.
  • We recently modified vanity to let us do splits other than 50%/50% as we wanted to use it to test with a smaller / controlled sample
  • However, since vanity only measures one metrics per test, and since we often care about two or more metrics – we augment the reporting by combining vanity data with our event logs (HAVE METRICS ON EVERYTHING) to form our own views of the experiment results

❖ ❖ ❖

%d bloggers like this: