A Practical Guide

Using Bayesian Statistics to Evaluate an Intervention

Routine quality monitoring, A/B testing, and how to do it with the Bayesian Posterior Calculator

This guide shows how to use the Bayes calculator using simple Bayesian statistics. It is not a primer in Bayesian stats or stats in general.

The Bayes calculator helps you answer a fundamental question in intervention evaluation: “Is the intervention doing what I want it to do?”

This guide covers the principles behind ongoing monitoring (Part 1) and testing potential improvements (Part 2) and then covers step-by-step the use of the Bayes posterior likelihood calculator that you can use to work out the likelihood that a hypothesis is correct using the output from your favourite stats program (Part 3).

The one idea behind all of this

Classical statistical hypothesis testing asks: “Is there a statistically significant effect?” That gives you a p-value, which is easy to misread and says little about whether an effect is big enough to matter.

The Bayesian approach asks a more useful question: “Given the data so far, what is the probability that the true effect is at least as large as the smallest effect I care about?” The answer is a number between 0% and 100% that anyone can understand — for example “There is a 93% probability that the new version improves outcomes by 20%.”

You supply three things

An outcome you care about (activity minutes, quit rate, retention…).
A threshold — the smallest effect that is genuinely worth having. This is a judgement, not a calculation.
Your measured result — the effect you observed, with its uncertainty (a standard error, a confidence interval, or a p-value).

The calculator returns the probability that the true effect clears your threshold.

Part 1 · Ongoing evaluation (routine quality monitoring)

Here you are monitoring an ongoing intervention and want continual reassurance that it is still working as expected: the statistical equivalent of a quality-control chart. Data arrive over time, and you re-check periodically.

Set it up once, before you start

Choose your outcome and how it is measured. Keep it stable so results stay comparable over time.
Choose your threshold — the smallest worthwhile effect. If any improvement at all counts, the threshold is 0 (or 1 for odds/risk ratios). More often there is a level below which the effect is not worth the cost or effort; use that.
Choose your action rules in advance, e.g. “treat as working if the probability reaches 95%; treat as not worth continuing if it falls to 10%; otherwise keep collecting.” Deciding these up front stops you moving the goalposts later.

Then, at each review point

Pool all the data collected so far and obtain the effect estimate with its standard error (or a confidence interval, or a one-tailed p-value).
Enter it into the calculator with your threshold.
Read off the probability that the true effect exceeds the threshold, and act according to your pre-set rules.

Worked example — a continuous outcome

An app aims to increase physical activity. You decide the smallest worthwhile effect is 5 extra minutes per day (threshold = 5). After a month, pooling all users to date, the observed mean difference versus a comparator is 7.2 minutes, with a standard error of 2.5.

In the calculator: Mean Difference tab → enter mean difference 7.2, SE 2.5, threshold 5.

Result: about an 81% probability that the true effect is at least 5 minutes a day. Encouraging, but short of a 95% action bar — so under the rules above you would keep collecting data and re-check.

Why re-checking as data accumulate is fine

With repeated significance tests, looking again and again inflates the chance of a false positive finding, so people invent complex corrections. A Bayesian posterior probability does not have this problem in the same way: it is always a valid summary of what the data say so far, so you may look as often as you like.

But don’t fool yourself

Being able to look freely is not a licence to stop the moment the number looks good. Two simple safeguards keep you honest:

Set the threshold at the smallest effect that genuinely matters — not at zero. A meaningful threshold is much harder for random noise to clear.
Consider a mildly sceptical prior (Part 3). It pulls early, noisy results towards “no effect” until real evidence accumulates.

Both make the probability harder to inflate by chance alone.

Part 2 · Testing improvements

Now you have a proposed improvement. Call the current version A and the new version B. The question is not simply “is B different?” but “is B better by an amount worth whatever the cost of switching might be?”

Run the test cleanly

Randomly allocate users or occasions to A or B, so the two sets of numbers are comparable.
Pre-specify one primary outcome and the threshold before looking at results.
Let it run until you have a reasonable amount of data, then analyse the difference between B and A.

Turn the result into a decision

Your A/B platform (or a simple analysis) will give you the difference between B and A: a difference in means for a numeric outcome, or an odds ratio or risk ratio for a yes/no outcome, each with a confidence interval. Feed that into the calculator with your threshold.

Worked example — a yes/no outcome

You test a new reminder (B) against the current one (A) for 4-week retention on an app. Your analysis reports an odds ratio of 1.28 for B versus A, with a 95% confidence interval of 1.05 to 1.56. You decide switching is only worthwhile if the odds of retention rise by at least 10% — a threshold OR of 1.10.

In the calculator: Odds Ratio tab → enter OR 1.28, then the lower and upper CI (1.05 and 1.56) so it derives the standard error → threshold 1.10.

Result: about a 93% probability that B beats A by a worthwhile margin (an OR of at least 1.10). Just short of a 95% bar — so if switching is cheap and low-risk you might adopt anyway; if it is costly, run a little longer.

A note on being appropriately sceptical

Most tweaks make little difference, and small samples throw up flukes. A mildly sceptical ‘prior’ — centred on “no difference” with modest uncertainty — encodes that realism. It tempers over-excitement from a lucky early result while still letting a genuinely strong effect come through. If you prefer to let the data speak entirely for themselves, leave the prior off (the default); the result is then equivalent to the standard one-tailed test, but reframed as an intuitive probability.

Part 3 · Using the Bayes calculator

Open the calculator in a new tab ↗ and follow along with the steps below.

Step 1 — Pick the tab that matches your outcome

Across the top are four tabs. Choose by the kind of outcome you have:

Your outcome looks like…	Example	Use this tab	Threshold means…
A number / score	minutes of activity, weight, mood score, £ spent	Mean Difference	smallest difference worth having (0 = any effect is worth having)
Yes / no, per person	quit / not, retained / not, clicked / not	Odds Ratio or Risk Ratio	1 = no effect; e.g. 1.2 = 20% higher odds or risk; 0.8 = 20% lower odds or risk
Strength of a link	how strongly app use tracks with outcome	Correlation	0 = any association; 0.1 = at least a small one

For a yes/no outcome you can use either Odds Ratio or Risk Ratio; risk ratio (e.g. “retention was 1.15 times as likely”) is often the more intuitive to explain.

Step 2 — Enter the output from your stats program

Each tab offers two input methods; use whichever matches the numbers you have obtained from your stats program.

Estimate + SE / CI — enter the observed effect and its standard error. If you only have a 95% confidence interval, type the lower and upper limits into the CI fields and the calculator derives the standard error for you.
One-tailed p-value — enter the observed effect and a one-tailed p-value in the direction of that effect. Use this when a report gives you a p-value but not a standard error.

Two things to get right

The p-value must be one-tailed and in the direction your effect actually went. If you only have the usual two-sided p-value and the effect is in the expected direction, halve the p value.
Ratios are handled on the log scale automatically. Just enter odds ratios, risk ratios and their confidence limits on their natural scale (e.g. 1.28, 1.05, 1.56); the calculator does the rest.

Step 3 — Set the threshold

The threshold is the heart of the method: the smallest effect worth caring about. For a numeric outcome, 0 means “any effect at all”; set it higher if you want to check that an effect is bigger than some worthwhile amount. For odds and risk ratios, 1 means “no effect”; set it to, say, 1.2 (a 20% increase) or 0.8 (a 20% reduction). For a correlation, a threshold of 0 means you are interested in any positive association.

Step 4 — Add a ‘prior’ (optional)

Leave this off to let just these data speak for themselves. Tick Use informative prior when you have genuine prior knowledge, say from a previous data set. You give the prior a mean (your best guess before seeing the data — often 0, or 1 for ratios, meaning “no effect”) and an SD (how sure you are — smaller means more sceptical). The result then blends your prior with the data. There is also a tick-box to draw the flat (non-informative) prior on the plot for comparison.

Step 5 — Read the result

The big percentage is the headline: the probability that the true effect exceeds your threshold.
The coloured chip gives a plain-language reading of how strong that is.
The detail rows show the posterior mean, SD and median (your best single estimate and its uncertainty after the analysis), and an “equivalent one-tailed p-value” for readers who still think in those terms.
The distribution plot shows the full range of plausible values for the true effect; the dashed line is your threshold. The share of the curve to the right of the dashed line is the headline probability.

Optional — the sliders

Tick Add sliders to alter your inputs and watch the probability and the curve update live. This is excellent for building intuition: for instance, seeing how much more data (a smaller standard error) you would need to push the probability past your action threshold, or how sensitive the answer is to where you set the threshold.

Reading the numbers sensibly

A rough reading of the headline probability

95% or more — strong evidence the effect clears your threshold; act with confidence.
80–95% — probably worthwhile, but you may want more data before a costly decision.
50–80% — leaning positive but uncertain; keep collecting.
Below 50% — the effect more likely falls short of your threshold than clears it.

These bands are conventions, not laws. Choose action thresholds that fit the stakes of your decision.

Assumptions and limits (worth knowing)

One comparison, one outcome at a time. If you monitor several outcomes at once, a couple may cross a high bar by chance; interpret accordingly.
It assumes your estimate follows an approximately ‘normal’ distribution (on the log scale for ratios, the Fisher-z scale for correlations). This holds well for the effect sizes and sample sizes typical of app evaluation.
Quality in, quality out. The maths cannot rescue a biased comparison. Randomise A/B tests where possible, and keep monitored outcomes measured consistently.
The threshold is a judgement. Most of the method’s value comes from choosing it thoughtfully and in advance.

The mathematics, in one box (skip if you like)

For the curious

The calculator uses a normal model. With a non-informative prior, the probability that the true effect θ exceeds threshold τ is:

P(θ ≥ τ | data) = 1 − Φ( (τ − estimate) / SE )

where Φ is the standard normal cumulative distribution. With an informative prior N(μ₀, σ₀²) the posterior is also normal, combining prior and data in inverse-variance (precision) proportion:

posterior mean = (μ₀/σ₀² + estimate/SE²) / (1/σ₀² + 1/SE²)
posterior SD = 1 / √(1/σ₀² + 1/SE²)

The key equivalence: with a flat prior and a threshold of 0 (or 1 for ratios), the posterior probability of a positive effect equals 1 minus the one-tailed p-value. The Bayesian framing simply generalises this to any threshold and lets you fold in prior knowledge.

Quick checklist

Decide the outcome, the threshold, and your action rules before looking at data.
Pick the matching tab (numeric → Mean Difference; yes/no → Odds or Risk Ratio; link → Correlation).
Enter your result (estimate + SE/CI, or a one-tailed p-value) and the threshold.
Add a sceptical prior if early or small-sample data could mislead (optional).
Read the probability, check it against your action rules, and decide.

Glossary

Terms marked with a dotted underline in the text link here. Use your browser’s Back button, or the “back” link in each entry, to return to where you were reading.

Odds ratio (OR)

A way of comparing the odds of an outcome between two groups or conditions. The odds of an outcome are the probability that it happens divided by the probability that it doesn’t (so a 25% chance is odds of 0.25 / 0.75 = 1 to 3). The odds ratio divides the odds in one group by the odds in the other.

OR = 1 means the odds are the same in both groups; greater than 1 means higher odds in the first; less than 1 means lower. Note that an odds ratio is not the same as “how many times more likely” — for common outcomes it exaggerates the difference compared with the risk ratio. The calculator works with it on the log scale, because log(OR) behaves more like a normal distribution.

Risk ratio (RR, “relative risk”)

Despite the name, the “risk” here simply means the proportion (or probability) of cases showing the outcome — it need not be anything bad. The term is a hangover from epidemiology, where the outcomes of interest were usually risks of disease, so any probability of an event came to be called a “risk”.

A risk ratio is therefore really just the ratio of two likelihoods that happen to be labelled “risks”: the proportion with the outcome in one group divided by the proportion in the other. RR = 1 means no difference; 1.5 means the outcome is 1.5 times as likely; 0.8 means 20% less likely. It is usually easier to explain than an odds ratio (“1.5 times as likely”). Like the OR, it is analysed on the log scale.

Mean difference

For an outcome measured as a number — a continuous outcome such as minutes of activity, a mood score, or pounds spent — the mean difference is simply the average value in one group or condition minus the average in another (or the average amount of change). A positive or negative value shows the direction; zero means no difference. This is the natural effect measure for numeric outcomes.

Correlation (Pearson’s r)

A measure of how strongly two numeric variables move together, running from −1 (a perfect inverse relationship) through 0 (no straight-line relationship) to +1 (a perfect positive one).

Here we use Pearson’s correlation coefficient, which captures the strength of a straight-line (linear) association. It says nothing about non-linear patterns and can be pulled about by extreme values, so it is best read alongside a scatterplot. The calculator handles it internally on the Fisher-z scale, where its sampling distribution is closer to normal.

Prior

In Bayesian analysis, what you believe about the size of the effect before taking the current data into account, expressed as a probability distribution (a best guess plus how uncertain you are about it).

A non-informative (or flat) prior deliberately expresses no opinion and lets the data speak entirely for themselves. An informative prior encodes real prior knowledge — for example results from an earlier study — or a deliberate dose of scepticism (typically centred on “no effect”). The prior is combined with the data to produce the posterior.

Posterior likelihood

What you believe about the effect after combining your prior with the observed data: the updated probability. In this calculator the headline output is a posterior probability — the probability, given the data (and any prior), that the true effect exceeds your threshold, i.e. that your hypothesis is correct.

A note on wording: statisticians usually call this the posterior probability. The word likelihood on its own is normally reserved for a related but different ingredient — how probable the observed data are for each possible effect size. Prior × likelihood, suitably scaled, gives the posterior. Nothing in how you use the calculator depends on this distinction.

Standard error (SE)

A measure of how much an estimate would jump around from one sample to the next: in effect, the standard deviation of the estimate itself. A smaller SE means a more precise estimate, and usually comes from more data.

The SE drives both the width of a confidence interval and the spread of the posterior. As a rough guide, a 95% confidence interval runs from about the estimate minus two SEs to the estimate plus two SEs.

Confidence interval (CI)

A range of values, calculated from the data, designed to contain the true effect with a stated level of confidence — almost always 95%. Loosely, if you repeated the study many times, about 95% of the intervals produced this way would contain the true value.

A narrow interval means a precise estimate. If a 95% CI for a difference excludes 0 (or, for a ratio, excludes 1), the result is “statistically significant” at the two-sided 5% level. The calculator can work out the standard error for you from a 95% CI.

One-tailed p-value

The probability of getting an effect at least as large as the one you observed, in one specified direction, if the true effect were actually nil. It suits a directional question such as “does the app increase activity?”.

In this calculator you enter a one-tailed p-value pointing in the direction of your observed effect.

Two-tailed p-value (also “two-sided”)

The probability of getting an effect at least as extreme as the one observed in either direction, if the true effect were nil. This is the usual default reported by statistics software.

For the symmetric tests used here, the one-tailed p-value in the observed direction is simply half the two-tailed value — so if you only have the two-tailed number and the effect points the way you expected, halve it before entering it.