what's next?

Isolate The Test


❖ ❖ ❖

We don’t want this test to impact other tests


Isolate the test. Review where the new test will go in all the various user flows (i.e. first-time user, returning user, user changing their profile, etc.) Consider when several tests are together, how they multiple the number of alternative paths. Make sure the new test doesn’t overlap any current tests and doesn’t change any data that current tests depend on.

  • When reviewing current tests, it may be useful to consider the current test as having a “begin” and an “end” in the user flow
    • “begin” is where it’s decided a user is in the test and which alternative they will be given
    • “end” is where success is measured
  • You can skew the results, by doing something before the “begin” of an existing test that changes the conditions in the middle of that existing test.
  • Once a test has started, if you change who gets to the “begin” of a test then it may mess with results
  • Once a test has started, if you do something that changes any of the data used to decide in whether a user is in a test, or whether they’ve succeeded with the test, then it may mess with the results
  • Add a new test in the middle of an existing test – where one or both alternatives add a new step for the user. Then the results of the existing test will be polluted.

❖ ❖ ❖

now you’re ready to implement the change, BUILD THE TEST and release it via CONTINUOUS DEPLOYMENT

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: