Experiments

How it works

• You assign, we observe. ClickClacks doesn’t split traffic or deploy anything. Your code or flag tool sends an exposure event that names the variant.
• First exposure wins. A person’s variant is the one in their first qualifying exposure. If they later see another arm, they stay in the first.
• No variant, no count. Exposures without the variant property are left out.

Already running tests? The empty Experiments page scans the last 7 days for exposure-like events and lists tests already running, with a confidence of High confidence, Likely or Possible. Click Set up analysis on one to start from it.

Set up an experiment

Click New experiment. The setup has five steps:

1. What you’re testing: a name, such as Pricing page layout, and an optional hypothesis.
2. Exposure: choose Detected, Your tool or Custom. Then set who is in the test (Anyone who fires an event), optionally only when a property names this test, and which property holds the variant. A strip checks the last 24 hours and says what it has seen.
3. Variants: every value seen is listed with its count. Rename arms, merge aliases into one arm, ignore junk values, and pick the control. You can have up to 8 arms. Set the Expected split to Even split or Custom shares.
4. Success metric: They fire an event within a number of days (1 to 90, 7 by default), and whether it should increase or decrease. Click Add a guardrail for metrics that must not get worse. Set the smallest change worth finding, for example 2 points, at 90%, 95% or 99% confidence.
5. Review: check everything, then click Start experiment. Or Save draft to finish later.

While you set up, Live exposure check shows arm counts and warns about problems such as unmapped values, people who saw more than one arm, or an uneven split. Before you start estimates how many people you need and when you are likely to finish. Halving the change you want to catch roughly quadruples the wait.

Sending exposures

Pick Your tool and ClickClacks expects these names:

From your own code, send one event when the person sees the variant:

window.clickclacks('event', 'experiment_viewed', {
  experiment: 'pricing-layout',
  variant: 'annual-first',
})

Send the variant as text. True or false values arrive as "true" and "false". More on sending events in Events and identify.

Always-valid results

Classic A/B statistics assume you look at the result once, at a finish date you fixed in advance. Peek every day and stop when it looks good, and you will crown false winners.

ClickClacks uses always-valid intervals everywhere. The interval holds at every look, so checking daily doesn’t spoil the result. Until the interval clears zero, the result stays Not enough data for a conclusion. There is no fixed finish date: the planning estimate says roughly how long it may take.

Reading results

• With two arms, the headline says Treatment improves the primary metric, Treatment is worse on the primary metric, or Not enough data for a conclusion. It shows the observed difference in points and percent, and the interval as a bar. It is conclusive when the bar doesn’t cross zero.
• With more arms, each treatment is compared with control. You see Winner, Control wins or No treatment beat control. Each comparison uses a stricter threshold so the confidence you chose holds for the whole experiment.
• Guardrail harm detected means a guardrail got worse. A good primary result doesn’t cancel that risk.
• The Planning estimate shows how many people you have out of about how many you need.

Results count from the moment you start, across all sources, within each experiment’s own start date and conversion window, not the date range in a filter bar.

Broken assignment (SRM)

If the arms’ sizes differ from the split you expected by more than chance allows (p < 0.001), the page says Assignment looks broken — results are not trustworthy and hides the results. This is a sample ratio mismatch. It usually means a bug: one arm fails to send its exposure, redirects lose people, or bots land in one arm. Fix the cause. Don’t read around it.

Stopping and deciding

• Record decision: pick the selected variant and confirm that you understand this records a decision and doesn’t deploy anything. Observation ends now. Roll out the change in your own flag tool.
• Stop test: ends observation without a decision.
• Once running, the exposure, metrics and window are locked. You can still rename, edit the hypothesis and fix the variant mapping. Every change is recorded under Setup changes.

Tips and gotchas

• No exposures yet? Send one from your app with the snippet. It appears in the mapper within a minute.
• Typos split your arms. If Treatment and treatment both show up, merge them in Variants.
• From a friction finding, Set up an experiment for this fix starts a draft named after it.

Recipes