Probability to Be Best (P2BB)

Probability to Be Best (P2BB) is a statistical metric used in A/B testing to estimate how likely it is that a particular variant is the best-performing option in a test.

P2BB is calculated using Bayesian statistics, which take into account both the observed results and the uncertainty in the data. It helps you make informed decisions about which variant to choose based on probability, not just raw conversion rates.

Why it matters

Unlike simple metrics like conversion rate, P2BB accounts for uncertainty and variation in your data. It gives you a clearer signal about which variant is likely to win — especially helpful when results are close or data is limited.

How to interpret P2BB

  • Expressed as a percentage

    A higher percentage means a higher likelihood that the variant is the best performer.

    Example: P2BB of 80% means there’s an 80% chance that this variant is better than the others.

  • Compares variants directly

    You can directly compare which variant is leading.

    If Variant A has a P2BB of 70% and Variant B has 30%, Variant A is more likely to be the better choice.

  • Supports better decision making

    Use P2BB to decide which variant to keep, ship, or iterate on — especially when combined with other business data.

When to determine a winner

P2BB is dynamic

P2BB updates as new data comes in. Early on, it may fluctuate significantly. As more traffic flows through the test, the values stabilize.

It’s not absolute

P2BB shows probability, not certainty. It’s a guide, not a guarantee. Consider it alongside other metrics like lift, conversions, and sample size.

Use with other metrics

P2BB is most effective when paired with:

  • Conversion rate trends

  • Traffic volume

  • Statistical confidence

  • Business impact

Wait for 95% confidence

We recommend waiting until a variant reaches a P2BB of 95% or higher before calling it a winner. This helps avoid premature decisions based on randomness or small sample sizes.

Example

You’re testing two homepage designs:

  • Variant A reaches a P2BB of 75%

  • Variant B has a P2BB of 25%

This means Variant A currently has a 75% chance of being the better-performing design — but it’s not yet conclusive.

You should continue the test until one variant reaches at least 95% P2BB for high confidence in the result.

PreviousWhat does the custom script do?NextHow to setup Optibase

Last updated