👑Probability to Be Best (P2BB)

What is P2BB?

Probability to Be Best (P2BB) is a statistical measure used in A/B testing to evaluate the likelihood that a particular variant (like Variant A vs. Variant B) is the most effective based on collected data. It is calculated using Bayesian statistics, which consider the observed results and the uncertainty inherent in testing.

How to Interpret P2BB

• Expressed as a Percentage: P2BB is shown as a percentage. A higher percentage means a greater likelihood of that variant being the best performer in the test.

• Direct Comparison: P2BB allows for a direct comparison between variants. For example, if Variant A has a P2BB of 70%, it suggests there's a 70% chance that Variant A is more effective than other variants in the test.

• Informed Decision Making: Use P2BB to guide decisions on which variant to implement. A variant with a substantially higher P2BB is usually the preferred option.

When to Determine a Winner

• P2BB is Dynamic: Keep in mind that P2BB values can change with ongoing data collection. Early in testing, the metric might be less stable and can become more reliable as more data accumulates.

• Not Absolute: P2BB is an indicator of likelihood, not a definitive measure of success. It should be interpreted within the context of the test and alongside other business metrics.

• Use with Other Metrics: For a comprehensive evaluation, consider P2BB in conjunction with other metrics and overall business objectives.

• Wait for 95% Confidence: It is advisable to wait until a variant achieves a P2BB of at least 95% before declaring it as the winner. This high threshold ensures a greater level of confidence in the test results, reducing the likelihood of making a decision based on insufficient data or random fluctuations.

Example

In an A/B test comparing two website designs, if Design A reaches a P2BB of 75%, it indicates a 75% probability that Design A is better than Design B based on the current data. However, to make a more confident decision, wait until either design achieves a P2BB of 95% or more.