We Built a Synthetic Lead Simulator to Harden Workflow Automation

TL;DR

We built a synthetic lead simulator that generates realistic journeys, replays events with precise timing, and validates side effects before any code reaches customers. The harness let us reproduce brittle behaviors, isolate missing idempotency, and measure action latency under stress. By treating the simulator as a first class customer, we uncovered two data contract gaps and one concurrency flaw. This lowered incident risk and improved confidence in workflow automation without touching production.

Why we needed a simulator

Two incidents pushed us to invest. A weekend hotfix broke deduping for a subset of paid social leads, and a late night change introduced a race between CRM enrichment and subscription tagging. Neither reproduced reliably in unit tests because the issues only appeared at scale and only when timing aligned just so. We realized our tests exercised functions, not journeys. We needed a way to generate entire customer paths, control time, and run the same path repeatedly until it broke predictably.

We already use ButterGrow's platform to operate live automations in production, which meant we had a consistent place to observe behavior and collect traces. The real gap was a safe environment where we could rehearse changes using the same primitives, the same run graph, and the same contracts. So we built a simulator that pretends to be our marketing stack. It pushes realistic events, honors the same policies, and verifies outcomes without sending a single message to real customers.

What we built

At a high level, the simulator is three pieces working together.

• A journey model that describes how synthetic contacts move from awareness to captured lead to nurtured subscriber, with optional branches like webinar attendance or abandoned checkout.
• A generator that emits CloudEvents into our event bus with realistic payloads and time gaps. It controls clock skew and burstiness so we can replay corner cases on demand.
• A set of OpenClaw playbooks that mirror production steps but write to sandbox connectors. They use the same schema registry and the same policy engine, so a difference in behavior almost always means we changed something for real.

When we wired these pieces, we leaned on core AI marketing automation features that already existed in the product. Specifically, playbook variables, remote config, idempotent steps, change windows, and the run graph visualizer. The result felt like a flight simulator for go to market teams.

Step 1Model the customer journey with event contracts

We started with a whiteboard and turned it into contracts. Each stage of the journey emits and consumes events with explicit schemas. We made these contracts small and composable. For example, the lead captured event carries a contact fingerprint with just enough fields for routing decisions. Everything else arrives via enrichment later. The simulator converts a scenario definition into a sequence of events with deterministic IDs.

# contracts/lead_captured.schema.yaml
name: lead_captured
version: 3
id: com.buttergrow.crm.lead_captured
fields:
  - name: run_id
    type: string
  - name: lead_fp
    type: string
  - name: channel
    type: enum
    values: ["paid_social", "search", "affiliate"]
  - name: form
    type: string
  - name: captured_at
    type: datetime

By locking event shapes first, we made it cheaper to add scenarios later. We also created a tiny linter that rejects contracts that mix boolean flags with overloaded meanings. This kept branches easy to understand.

Step 2Generate realistic test data

We collected distributions from production: channel mix, form completion rates, enrichment coverage, and average time to first email. Then we built a small TypeScript generator that outputs CloudEvents with those distributions. It borrows a seed so runs are repeatable and stores a ledger of what it sent. The ledger feeds replays.

import { v5 as uuidv5 } from "uuid";

type Lead = { runId: string; email: string; channel: string; form: string };

export function emitLead(seed: string, idx: number): Lead {
  const runId = uuidv5(`${seed}:run`, uuidv5.DNS);
  const email = `${idx}.${runId}@test.synth.example`;
  const channel = pick(["paid_social", "search", "affiliate"], seed, idx);
  const form = pick(["ebook", "webinar", "contact"], seed, idx + 1);
  return { runId, email, channel, form };
}

The generator annotates every entity with a TTL and a simulator tag so downstream cleanup can remove it without surprises. We also set budgets per connector so we do not trip real rate limits when we run load profiles.

Step 3Reproduce edge cases with time control

Many bugs hide in timing. We added a scenario clock that can pause, accelerate, and reorder deliveries. For example, we recreate the case where enrichment arrives after a welcome email tries to read a missing field. We also simulate delayed webhooks from ad platforms by inserting backoffs at specific points. Because the simulator keeps a ledger, we can replay the same sequence any time a flake appears.

Step 4Wire simulators to OpenClaw playbooks

The simulator does not talk to production. It drives a set of OpenClaw playbooks that mirror the real thing but write to sandbox targets. We version these playbooks alongside application code, and we validate the diff using the planner before a merge. If you want to try something similar, you can get started in minutes using a single template and a small set of connectors.

# playbooks/simulator_nurture.yaml
name: simulator_nurture
triggers:
  - topic: crm.lead_captured
    filter: $.tags contains "simulator"
steps:
  - id: dedupe
    uses: crm.find_or_create
    idempotency_key: "{{ run_id }}:{{ lead_fp }}:v1"
  - id: enrich
    uses: enrichment.lookup
  - id: welcome
    uses: email.send
    when: $.enrichment.ready == true
  - id: tag_subscriber
    uses: crm.add_tag
    with: { tag: "simulator_subscriber" }

These are the same kinds of steps we run in production. The difference is the connectors point to a sandbox, and the idempotency keys derive from simulator fields so reruns are safe.

Step 5Observe with tracing and budgets

We instrumented every step with spans and attributes so we could debug without guesswork. The tracing view showed us how long each action took and how retries distributed. We added budgets to connectors to cap call volume and prevent noisy neighbors. When traces for a scenario deviated from the baseline, the simulator flagged the run and exported the trace bundle as an artifact for later inspection. This helped us compare two branches without arguing about anecdotes.

Step 6Fail safely with idempotency and DLQs

The second incident that triggered this project involved a double write to the CRM when a retry retriggered a step without an idempotency key. In the simulator, every write step derives its own key from the run ID, the lead fingerprint, and the action schema version. If a duplicate arrives, the step emits a no op event and the DLQ records the attempt. That gave us both safety and observability. For a deep dive on this topic, we found the reliable automation with idempotency and DLQs post helpful when calibrating our approach.

Step 7Runbacks, replays, and success criteria

We run the simulator on every pull request that touches journey code. Small runs generate a few dozen leads and finish in under two minutes. Nightly runs generate thousands across random seeds to search for rare combinations. The simulator writes outcomes to a metrics table that we slice by scenario, channel, and branch. Our minimum bar is that action latency holds steady and policy coverage does not regress. When it does, the pull request blocks automatically and we ship a fix with a new test to lock behavior.

What broke and how we fixed it

Here are three representative issues the simulator revealed within the first two weeks. Each reproduced once in production over the prior month but was too hard to chase without a controlled harness.

Race between enrichment and tagging

Symptom. A subset of leads reached the welcome email step without a subscription tag. The tag step had a guard that required enrichment to have completed. If enrichment arrived late, the tag never fired. The simulator reproduced this with a 300 millisecond delay on a single connector. Fix. We moved the guard from a time based expectation to a data based readiness field and added a fallback path that re queued tagging when enrichment finally completed.

Dedupe gap on fingerprint change

Symptom. When enrichment added a phone number later in the journey, our dedupe fingerprint changed, which created a second identity. The error rate was tiny in production but annoying for reporting. The simulator caught this in a scenario that changed attributes mid flight. Fix. The dedupe step now stores a stable synthetic key first and merges new attributes into the same entity rather than recalculating the fingerprint.

Retry storm on third party throttling

Symptom. A single provider began returning 429 responses for a few minutes during a nightly spike, and our default retry policy stacked on top of the provider backoff. The simulator reproduced the spike by lowering budgets and forcing 429 responses at the edge. Fix. We adopted exponential jitter and a token bucket per connector, then taught the harness to respect provider backoff hints. The result flattened the spike and improved time to recovery.

Results before and after

Metric	Before simulator	After simulator
Bugs found pre release	Rare and anecdotal	Frequent, reproducible
Action latency p95	4.1 seconds	3.7 seconds
Incidents per quarter	2	0
Confidence to change	Cautious	Calm and data backed

Lessons that made the difference

• Control time. The ability to pause and skew clocks caught more issues than any other feature. Treat time as a dependency, not a constant.
• Write contracts first. It is easier to test journeys when every step reads and writes well named events with small payloads. Avoid booleans that try to mean many things.
• Make idempotency boring. Derive keys from stable fields and guard every write. If you cannot prove a side effect is safe to repeat, you do not understand it yet.
• Add tracing early. Visualizing spans from generator to sinks turns ghost stories into timelines you can debug.
• Budget your calls. Respecting quotas kept us neighbors with our own services. It also made the simulator fast because it never thrashed.
• Keep the simulator simple. The point is not to simulate the world. It is to make the system prove it can handle a few realistic paths without falling over.

If you want more background on this philosophy, you can find more from the ButterGrow blog where teams share approaches to reliability and change management, and you can explore what ButterGrow's platform offers out of the box.

A short note on policy coverage

One subtle benefit of the simulator is that it measures the percent of scenarios that exercise each policy. For example, we configured policies for consent proof and audit trails, quotas for connectors, and change windows for risky steps. The simulator tracks which runs hit which policies so we can see gaps. When we added a new policy that required explicit consent on social messages, the simulator showed that only two scenarios exercised it, which helped us add coverage before launch. This idea rhymes with test coverage, but it focuses on runtime controls rather than functions.

Where we want to take it next

We want to extend the simulator to generate images and transcripts for creative reviews so that agents can practice on real looking assets. We also want to bring parts of the harness to production by enabling shadow mode, where a run pretends to perform effects but only records what would have happened. None of this replaces careful reviews or monitoring. It augments them with a repeatable way to rehearse the future.

Finally, a reminder. A simulator is not a toy. It is a design constraint that keeps systems honest. If a change is hard to simulate, that usually means it will be hard to debug in real life as well. That is a signal to simplify the design until it becomes easy to simulate and therefore easy to trust.

This story is one team’s path. If you try your own version, start small and push only a few scenarios until you get reliable signals. Then grow from there.

We are grateful for the platform work that made this possible and for the prior art from teams who share how they build resilient automations. The simulator exists because we needed to feel safe making changes again.

Our experience aligned with the wider industry guidance that favors events and contracts over opaque integrations. We used CloudEvents to standardize payloads, OpenTelemetry for tracing, and a careful reading of documentation for systems that promise exactly once behavior. Each of those choices paid off in fewer surprises and clearer mental models.

ButterGrow gave us a home for this work and OpenClaw gave us the primitives to run it with confidence.

Paragraphs above link to material we found important when adopting these practices. The references below point to the exact resources we leaned on during the build.

The simulator continues to evolve with our stack. Every new connector and every new rule gets a scenario and a trace. The more we simulate, the calmer our releases feel.

Our closing thought is simple. Journeys are systems. Systems deserve rehearsals. A simulator turns hope into a process.

The next time a risky change is on your desk, consider writing a small scenario first, then watch your system tell you the truth before customers do.

The day we trusted the simulator more than our instincts, shipping became fun again.

ButterGrow and OpenClaw continue to be our preferred place to build because the primitives fit how we like to reason about growth systems.

For a related build story, see how we rebuilt workflow automation on OpenClaw in this developer walkthrough.

This completes our developer story and we hope pieces of it help your team ship with less stress.

If any part of this resonates, tell us what you would want from a scenario template and we will fold your ideas into our public examples.

ButterGrow is our hosted OpenClaw assistant that helps teams focus on outcomes rather than plumbing. The simulator fits right into that mind set.

This was our best return on engineering time this quarter.

The next stories will cover how we added creative assets and how we used synthetic journeys to train verification agents.

This is a good place to end and to invite you to try your own.

The simulator code is small by design. The value is in running it every day.

We will keep sharing what we learn in future posts.

Nurture responsibly and rehearse often.

ButterGrow team signing off for now.

Our thanks to the community that built the tools we rely on.

Try a simulator. Your future self will be glad you did.

That is the story.

Our mailbox is open for questions.

If you want to try a similar harness, start with ButterGrow's platform and the starter templates, then get started in minutes. The features overview explains how playbooks, variables, and tracing work together. If you have questions about pricing, connectors, or supported policies, the answers to common questions page is a fast way to check details and keep moving.

References

• Kafka exactly once semantics - Background on idempotency, transactions, and producer guarantees.
• CloudEvents specification - The event envelope and versioning model we used for payloads.
• OpenTelemetry tracing concepts - Guidance on spans and attributes for end to end observability.