PA Data Analytics
  • Portfolio project
  • Synthetic dataset
  • Marketing Analytics

Email A/B test analysis: evaluating two campaign variants

A portfolio A/B test evaluation comparing two email variants across the full funnel, from opens to purchases, with statistical significance testing.

Project type
Portfolio project
Dataset
Synthetic dataset
Dataset provenance
A synthetic email-campaign dataset (25,000 customer records over a 90-day campaign) generated for analytical practice.
Client status
Demonstration analysis; does not represent a PA Data Analytics client.
VARIANT A VS VARIANT BVariant BVariant AOpen rate, Variant A: 23.99%23.99%AOpen rate, Variant B: 36.66%36.66%BOpen rateClick rate, Variant A: 4.24%4.24%AClick rate, Variant B: 10.15%10.15%BClick rateConversion rate, Variant A: 0.59%0.59%AConversion rate, Variant B: 1.76%1.76%BConversion ratePortfolio project · Synthetic dataset · each panel has its own scale
Chart drawn from the project's own outputs · Synthetic dataset · portfolio project.
View the data behind this chart
MetricVariant AVariant B
Open rate23.99%36.66%
Click rate4.24%10.15%
Conversion rate0.59%1.76%

01Business question

When a marketing team tests a new email variant, it needs to know whether the variant genuinely performs better across the funnel, or whether an apparent difference is noise, and whether any improvement comes at a cost such as more unsubscribes.

02Dataset & context

Dataset: Synthetic / Demonstration Dataset. This portfolio analysis uses a synthetic email-campaign dataset generated for analytical practice: 25,000 customer records over a 90-day campaign, split between Variant A (12,639 customers) and Variant B (12,361). It does not represent a PA Data Analytics client.

03Approach

Define success criteria before looking at results: a statistically significant improvement in at least two of four primary metrics, and no significant increase in unsubscribes. Then evaluate each stage of the funnel separately.

04Methodology

  • Funnel metrics: open rate, click rate, conversion rate and revenue per send.
  • Two-proportion z-tests with 95% confidence intervals for open, click and conversion rates, at a 5% significance level.
  • Mann-Whitney U and Welch's t-tests for revenue per send, because revenue was heavily skewed.
  • Breakdowns by customer segment, device, subject line and send time.

05Modelling & analysis

Dataset findings:

  • Open rate: 23.99% for Variant A vs 36.66% for Variant B.
  • Click rate: 4.24% vs 10.15%.
  • Conversion rate: 0.59% (74 of 12,639) vs 1.76% (218 of 12,361). The 95% confidence interval for the difference was 0.91 to 1.45 percentage points.
  • Revenue per send: $4.56 vs $18.33.
  • Unsubscribe rate: 1.50% for both variants.

All four primary differences were statistically significant (p < 0.0001). Purchase counts are small (74 and 218), which is why the confidence interval is reported alongside the p-value.

06Results within the dataset

In this dataset, Variant B met the pre-defined success criteria on every primary metric without increasing unsubscribes. These are findings from synthetic data. They are not revenue earned by any business, and no revenue projection is presented here.

07What this analysis demonstrates

A complete testing workflow: defining success criteria in advance, testing each funnel stage with the appropriate statistical test, handling skewed revenue data, and reporting uncertainty rather than a single headline lift.

08How a business could use this analysis

  • Test email, website or campaign changes before a full rollout, so decisions rest on evidence rather than opinion.
  • Decide success criteria and sample sizes up front, and avoid stopping tests early because an interim result looks good.
  • Check guardrail metrics such as unsubscribes alongside the headline metric.

09Related service

This project demonstrates methods used in our marketing analytics services.

10Apply this to your data

If you are facing a similar question, we can look at what your data can support and what a useful answer would look like.

Start a ConversationBook a Meeting

More case studies

SHARE OF REVENUE CREDITShapleyLast ClickPaid SearchPaid Search: 35.6% of credit under Last ClickPaid Search: 41.9% of credit under Shapley35.6%41.9%DirectDirect: 8.4% of credit under Last ClickDirect: 51.9% of credit under Shapley8.4%51.9%EmailEmail: 18.9% of credit under Last ClickEmail: 0.0% of credit under Shapley18.9%0%Paid SocialPaid Social: 15.0% of credit under Last ClickPaid Social: 6.2% of credit under Shapley15%6.2%InfluencerInfluencer: 7.6% of credit under Last ClickInfluencer: 0.0% of credit under Shapley7.6%0%Portfolio project · Synthetic dataset
  • Portfolio project
  • Synthetic dataset
  • Marketing Analytics

Comparing five attribution models to understand channel contribution

A portfolio analysis that builds five attribution models on the same customer journeys, to show how much a channel's credit depends on the model chosen.

Synthetic dataset: 11,292 touchpoints · 3,500 customers · 5 models

Read the attribution case study
MONTHLY REVENUEModel-fittedActual$0.6M$1.0M2023-01: actual $780,659, model-fitted $777,9402023-02: actual $756,909, model-fitted $756,3382023-03: actual $724,723, model-fitted $728,0512023-04: actual $705,401, model-fitted $705,1562023-05: actual $945,021, model-fitted $952,0832023-06: actual $752,169, model-fitted $730,9202023-07: actual $873,979, model-fitted $891,5472023-08: actual $791,576, model-fitted $790,1582023-09: actual $729,243, model-fitted $724,5182023-10: actual $894,399, model-fitted $913,6452023-11: actual $933,780, model-fitted $924,6182023-12: actual $948,259, model-fitted $952,0402024-01: actual $846,935, model-fitted $858,9172024-02: actual $794,305, model-fitted $805,7142024-03: actual $766,408, model-fitted $763,6472024-04: actual $948,479, model-fitted $936,6722024-05: actual $789,934, model-fitted $797,6692024-06: actual $787,235, model-fitted $784,7892024-07: actual $928,345, model-fitted $942,9262024-08: actual $826,781, model-fitted $803,0612024-09: actual $963,461, model-fitted $963,2772024-10: actual $752,190, model-fitted $763,5462024-11: actual $978,515, model-fitted $975,4352024-12: actual $1,004,106, model-fitted $980,147Jan 2023Jan 2024Dec 2024Portfolio project · Synthetic dataset · R² 0.957 in-sample
  • Portfolio project
  • Synthetic dataset
  • Marketing Analytics

Marketing mix modelling and budget allocation analysis

A portfolio marketing mix model that estimates each channel's contribution to revenue from two years of weekly data and tests how a fixed budget could be reallocated.

Synthetic dataset: 104 weeks · 4 paid channels · R² 0.957 (in-sample)

Read the marketing mix modelling case study

Let's turn your data into your next decision.

Tell us what you are trying to understand: marketing performance, customer retention, forecasting or reporting. We will reply with practical next steps.

Prefer email? [email protected]

Start a Conversation Book a Meeting