When should I pick epsilon greedy over Thompson sampling for marketing experiments?

Use epsilon greedy when you want a simple, robust baseline with a fixed exploration rate and stable traffic patterns. Choose Thompson sampling when reward variance is high and you want faster adaptation to winners. Both should log impressions and rewards per arm so you can audit allocation decisions later.

How do I define a reward for bandit tests in email subject line experiments?

Use a binary reward such as 1 for an open and 0 for no open within a defined window, or a weighted reward like 1 for an open plus 0.2 for a click. Keep the window consistent across arms and exclude bounced addresses. Always record exposure time to avoid survivorship bias.

What sample size is enough for bandit testing in a weekly campaign?

Bandits do not need a pre fixed sample size, but you still need minimum exposure to avoid premature convergence. A practical rule is at least a few hundred exposures per arm before reducing exploration, with a cap on minimum epsilon such as 0.05. For low volume, run longer windows or fewer arms.

Can I use bandits for pricing or only for creative variants?

You can apply bandits to pricing, creatives, and timing, but define guardrails like minimum margin, inventory constraints, and geo restrictions. Start with narrow price bands and switch to contextual bandits only after you validate a safe reward signal and constraints in logs.

How do I report results from a bandit instead of a traditional A/B test?

Report cumulative regret, allocation over time, and estimated lift versus a holdout or a synthetic baseline. Include confidence intervals from posterior samples when using Thompson sampling. Stakeholders should see how traffic shifted toward stronger arms and what the expected business impact is over the same horizon.

What logging is mandatory to keep the experiment auditable?

Log the arm presented, the policy parameters at decision time, a unique user or session key, the timestamp, and the observed reward within the chosen window. Store these in an append only table so you can reproduce allocations for compliance reviews and to run offline policy evaluation.

Bandit Testing for Conversion Optimization: A Practical Playbook for 2026

TL;DR

Bandit testing is a practical alternative to long A/B cycles when your goal is conversion optimization and you care about capturing lift sooner rather than only proving it later. Instead of splitting traffic evenly for weeks, a bandit allocates more impressions to options that perform better while still exploring new ideas. In this playbook you will learn when to use epsilon greedy, when Thompson sampling adapts faster, and how to set rewards and guardrails that reflect real business value. The examples map directly to ButterGrow and OpenClaw so you can deploy a working policy in a day.

What Bandit Testing Actually Solves

Classic A/B tests are great for clean inference, but they can be expensive in foregone upside while you wait for statistical significance. If one variant is clearly better, half your audience still receives the weaker option until the test ends. Bandit methods address this by shifting more traffic to better performing arms as data comes in, reducing regret while still learning.

At its core, a bandit policy chooses one of several actions, observes a reward, and updates beliefs so that future choices are more likely to pick winners. If you are new to the mathematics, start with a plain language summary of the multi armed bandit problem to ground the tradeoffs between exploration and exploitation. In marketing, the arms might be subject lines, thumbnails, send times, or call to action copy.

If you want context from automation tooling, our overview of AI marketing automation features explains how decision nodes, event sinks, and webhooks form the foundation for adaptive policies without extra infrastructure.

Algorithms You Can Use

Epsilon Greedy: Simple and Reliable

Epsilon greedy is the easiest policy to deploy. With probability epsilon, the policy explores by picking any arm uniformly at random. With probability 1 minus epsilon, it exploits by choosing the arm with the highest observed average reward. Because the exploration rate is explicit, this policy is predictable and easy to explain.

Advantages:

Transparent behavior with a single knob to tune.
Stable under moderate traffic and stationary rewards.
Works with binary rewards like opens or purchases.

Tradeoffs:

Fixed exploration can feel wasteful once a winner emerges.
Slower to adapt if performance shifts over time.

A useful long tail query to guide implementation choices is "epsilon greedy vs Thompson sampling for marketing," which captures how practitioners search for comparisons between these policies.

Thompson Sampling: Fast Adaptation From Uncertainty

Thompson sampling models uncertainty about each arm and samples from a posterior distribution to decide what to show next. Arms with better performance and tighter uncertainty get more traffic, but every arm is still sampled according to its probability of being optimal. For binary rewards like opens or purchases, a Beta Bernoulli model is common. You can read a clear overview of Thompson sampling on Wikipedia to understand why it often adapts faster than fixed exploration.

Advantages:

Aggressive allocation to promising variants when evidence is strong.
Naturally balances exploration by sampling from uncertainty.
Often yields higher cumulative reward in volatile environments.

Tradeoffs:

Slightly more complex to explain to non technical stakeholders.
Requires a reasonable prior or uninformative starting point.

Upper Confidence Bound: Optimism in the Face of Uncertainty

Upper Confidence Bound (UCB) algorithms select the arm with the highest upper bound on performance given the current estimate and uncertainty. This is another way to formalize exploration without random draws. UCB can work well when you can compute clean confidence bounds and want deterministic choices.

Bandits or A/B Tests: Choose With Intention

Use this table to decide quickly.

Situation	Prefer Bandit	Prefer A/B
You want faster capture of lift while learning	Yes	No
You need clean inference for a quarterly report	Maybe, with a holdout	Yes
Traffic is low and drift is likely	Yes, but with fewer arms	No, underpowered
You must compare two designs for a launch blog	Maybe, short bandit window	Yes

If your program requires a head to head comparison for compliance, you can keep a small holdout that receives a fixed control while the bandit allocates the rest. Optimizely's overview of multi armed bandit testing explains this hybrid approach in accessible terms.

Implementation Blueprint in ButterGrow and OpenClaw

This section shows how to wire a working policy that you can ship this week. The steps reference core product areas so you can map the design to real nodes and logs.

Step 1Define the Reward and Window

Pick a reward that matches business value and is observable soon. For email subject lines, use open within 24 hours. For a landing page hero, use click through within a session. If purchases are sparse, use an intermediate event such as add to cart with a weight. Document the window start and stop rules so you can reproduce results later.

Step 2Instrument Events and Bucketing

Configure event capture for impression, click, and conversion. Use a consistent user key for bucketing so the same person does not see different arms in a single session unless you intend to. ButterGrow supports this through event connectors and identity mapping. If this is your first time connecting sources, review what ButterGrow does on the feature set page for an overview of nodes that receive and evaluate events.

Step 3Choose Policy and Defaults

Start with epsilon greedy at epsilon equals 0.1. If you see high variance or rapid shifts, switch to Thompson sampling with a Beta prior of alpha equals 1 and beta equals 1 per arm. Cap minimum exploration at 0.05 so you always gather a trickle of data in case performance changes.

Step 4Wire the Decision Node and Arms

Create a decision node with one arm per variant. For email, arms map to subject lines. For paid media, arms map to creatives. The node should log the arm ID, the user key, and the policy parameters used at decision time.

Step 5Connect the Reward and Update Logic

When a reward event is observed within the window, send a success signal to the bandit node for that arm and user key. For epsilon greedy, update the running average reward. For Thompson sampling, increment alpha for a success or beta for a failure. Store these in a state table so decisions are consistent across processes.

Step 6Set Guardrails and a Holdout

Add guardrails such as minimum margin for pricing tests or maximum frequency per user for messages. Keep a 5 percent control holdout that receives a baseline variant so you can compute estimated lift over time. These settings make your program defensible in audits.

If you want a deeper channel specific walkthrough, our email marketing automation guide for 2026 shows how message pipelines, segmentation, and timing work together.

Practical Settings and Defaults

Start with 2 to 4 arms. More arms spread traffic thin and delay learning.
Use a daily reset for windows on channels with clear cadence like email, and session based windows for web.
Pause exploration during sensitive events such as a major product launch day.
For small programs, run one bandit at a time to keep analysis clean.

A helpful long tail phrase for this section is "bandit testing sample size for email subject lines." Even though bandits do not require a fixed sample size, the phrase reflects how marketers search for practical guidance on minimum exposure.

Measuring Impact and Guardrails

Stakeholders will ask how much lift you captured and whether the policy was safe. Provide three views:

Allocation over time, showing how traffic shifted toward strong arms.
Estimated cumulative reward versus the holdout or a synthetic baseline.
Confidence intervals on performance when using posterior draws.

Include guardrail metrics such as unsubscribe rate, spam complaints, or churn adjacent to the main chart. This makes it clear that the policy improves outcomes without trading away trust.

Common Pitfalls to Avoid

Using revenue as a reward with heavy seasonality and no normalization. Prefer intermediate events or a scaled reward.
Running too many arms with limited traffic, which stalls learning.
Forgetting to log policy parameters at decision time, which breaks audits.
Turning exploration to zero too early. Keep a small epsilon or equivalent.
Mixing windows across arms, which biases comparisons.

Example: Subject Line Experiment in 48 Hours

You manage a weekly newsletter with 120000 subscribers and want to improve opens. Create three subject lines and run epsilon greedy with epsilon 0.1 for the first send. After 24 hours, the policy shifts more traffic to the best line based on observed opens. In the second weekly send, you either introduce a fresh challenger or reduce exploration to 0.05 if lift looks stable. Report estimated cumulative opens versus the 5 percent holdout that receives your historical baseline.

If this scenario fits your roadmap, you can read answers to common questions about setup, pricing, and supported connectors before you ship the first policy.

Tooling and Data Model

Minimum tables to support a simple deployment:

Decisions: decision_id, user_key, arm_id, policy_name, policy_params_json, timestamp.
Impressions: decision_id, arm_id, timestamp.
Rewards: decision_id, arm_id, reward_value, timestamp.
Aggregates: arm_id, count, sum_reward, alpha, beta, updated_at.

This structure keeps online decisions fast while allowing offline analysis and replay. If you later adopt contextual bandits, you can add features to the decision record without changing the reward path.

Reporting and Handoff

Create a one page view each week: allocation chart, cumulative reward versus holdout, top line business impact, and guardrail metrics. Include a short note on policy settings and any changes you made. This makes it easy to hand off decisions across the team and to brief leadership without diving into raw logs.

To explore how this fits with broader automation, browse more from the ButterGrow blog and the overview of what ButterGrow does so you can connect experimentation with segmentation and scheduling.

A final long tail phrase to anchor search intent is "how to use multi armed bandits for conversion optimization." It captures the practitioner mindset and aligns with the steps above.

ButterGrow and OpenClaw give you the building blocks to wire this with minimal custom code. You can start small with a single decision node and grow into a program that continuously tests and learns across channels.

When you are ready to try this in your own stack, you can get started in minutes and connect the decision node template, an email connector, and a reward event without writing glue code.

This paragraph is your only call to action. If you prefer to explore first, review the features, connect a sample list, and follow the quickstart in the onboarding flow. ButterGrow runs on the hosted OpenClaw assistant so your team can ship policies without managing infrastructure.

References

Multi armed bandit problem - Background on exploration versus exploitation and the core objective of minimizing regret.
Thompson sampling on Wikipedia - Practical algorithm for binary rewards with a Beta Bernoulli model that adapts quickly.
Optimizely overview of multi armed bandit testing - Practitioner friendly explanation of hybrid approaches with holdouts.