… this helps complete TEST HYPOTHESIS WITH MINIMAL CHANGE
❖ ❖ ❖
We don’t want this test to impact other tests
Therefore:
Isolate the test. Review where the new test will go in all the various user flows (i.e. first-time user, returning user, user changing their profile, etc.) Consider when several tests are together, how they multiple the number of alternative paths. Make sure the new test doesn’t overlap any current tests and doesn’t change any data that current tests depend on.
- When reviewing current tests, it may be useful to consider the current test as having a “begin” and an “end” in the user flow
- “begin” is where it’s decided a user is in the test and which alternative they will be given
- “end” is where success is measured
- You can skew the results, by doing something before the “begin” of an existing test that changes the conditions in the middle of that existing test.
- Once a test has started, if you change who gets to the “begin” of a test then it may mess with results
- Once a test has started, if you do something that changes any of the data used to decide in whether a user is in a test, or whether they’ve succeeded with the test, then it may mess with the results
- Add a new test in the middle of an existing test – where one or both alternatives add a new step for the user. Then the results of the existing test will be polluted.
❖ ❖ ❖
now you’re ready to implement the change, BUILD THE TEST and release it via CONTINUOUS DEPLOYMENT
Steve Sanderson 