How to Create a New A/B Test

Once you understand how A/B Testing works, the next step is to create your own test.

This article walks you through every setting and option inside the “Create new A/B testing” page.

🧭 Accessing the A/B Test Creation Screen

1. From your Account Editor dashboard, go to Upsell Engine → A/B Testing.
2. Click “Create Test” (or “Create your first test” if none exist yet).

You’ll be taken to the setup page titled “Create new A/B Testing.”

At the top, you’ll notice the status toggle (Inactive/Active).

Leave it inactive while setting up — you’ll activate it after saving.

🧩 Step 1: Test Details

This is where you name your test and define what you’re testing.

Test Name

Give your A/B test a short, descriptive name.

Use clear identifiers such as:

• Type of offer (e.g. “Holiday Discount”)
• What’s being tested (e.g. “15% vs 20%”)
• Time period or target surface (e.g. “Checkout page”)

💡 Example:

Holiday Discount — 15% vs 20%

This makes it easy to find in the dashboard later.

💭 Step 2: Define Your Hypothesis

This section helps you set a clear goal for what you’re testing.

Under “What are you testing?”, write one simple, measurable statement.

💡 Example Hypothesis:

“Increasing the discount from 15% to 20% will lead to higher checkout conversion rates.”

A good hypothesis should:

• Clearly state the change being tested
• Mention what you expect to happen
• Be measurable (e.g., higher revenue, conversion, or AOV)

📏 Step 3: Configure Test Parameters

Now, you decide how success will be measured.

Under Test Configuration, select your success metric from the dropdown menu.

Available Metrics:

• Conversion Rate: Ideal if you’re testing layout or button text.
• Revenue: Use this for tests focused on pricing or discounts.
• Average Order Value (AOV): Useful when comparing bundle vs. single-item upsells.
• Click-Through Rate (CTR): Best for early engagement tests.

You can also enter a Projected Gain amount to estimate expected performance lift.

🗓️ Step 4: Set Test Duration

Specify how long your test should run.

💡 Tip: Shopify recommends running each test for at least two weeks for statistically valid results.

🧱 Step 5: Choose Surfaces & Offers

Next, select where this test will appear in your customer journey.

Click Select Surface, then choose one of the following:

• Checkout page
• After checkout
• Thank you page
• Order status page

⚠️ Note: A/B Testing requires at least two active offers on the same surface.

If you only have one, you’ll see a message:

​_“A/B testing requires a minimum of two offers per surface. Create an additional offer to proceed.”_

🔄 Step 6: Traffic Distribution

Decide how traffic is split between the two versions.

By default:

• 50% Control: Sees your current offer (no change).
• 50% Variant A: Sees the updated offer you’re testing.

Each visitor will be assigned one version and will continue to see that version throughout their session.

This ensures a fair comparison without crossover bias.

🚀 Step 7: Launch the Test

Once everything is set:

1. Review your setup details.
2. Click Save.
3. Switch the toggle from Inactive → Active to start collecting data.

Your test will now begin automatically on the selected start date.

✅ Example: Holiday Discount Test

🧠 Best Practices

• Test only one variable at a time (e.g. button color, discount %, product image).
• Avoid ending tests early — let them reach significance.
• Use consistent product and pricing settings across both versions.
• Always track metrics under the Analytics tab for deeper insights.

🔎 Next Article → Understanding A/B Test Results

In the next guide, we’ll explain:

• How to read variant performance metrics
• What makes a “winning” version
• How to apply the winner automatically

This ensures you can make confident, data-backed changes to your upsell strategy.

Updated on: 27/11/2025