How to Add New A/B Tests - Step-by-Step Guide

This guide walks you through adding new experiments using the Statsig experimentation system.

Overview: The 5-Step Process

1. Create Experiment in Statsig Console
   ↓
2. Configure in experiments.config.js
   ↓
3. Prepare DOM Elements
   ↓
4. Test Locally
   ↓
5. Deploy & Monitor

Step 1: Create Experiment in Statsig Console

1.1 Log into Statsig

Go to https://console.statsig.com and log into your project.

1.2 Create New Experiment

Navigate: Experiments → Create New Experiment

Settings: - Name: Use descriptive, snake_case naming (e.g., blog_hero_cta_text) - Type: Select "Experiment" (not "Feature Flag") - Allocation: Start with 50/50 split between variants

1.3 Define Parameters

Parameters are the values that change between variants.

Example: Text Change Experiment

Parameter Name: button_label
Type: String

Variants:
  - Control: "Read the Tutorial"
  - Treatment: "Start Learning Now"

Example: Layout Experiment

Parameter Name: layout_style
Type: String

Variants:
  - Control: "grid"
  - Treatment: "list"

Example: Number Experiment

Parameter Name: posts_per_page
Type: Number

Variants:
  - Control: 5
  - Treatment_A: 10
  - Treatment_B: 3

1.4 Set Success Metrics

Define what success looks like: - Primary: Click-through rate, conversion rate, etc. - Secondary: Time on page, scroll depth, bounce rate

1.5 Start Experiment

Keep it at 0% allocation initially. You'll ramp up after testing.

Step 2: Configure in Code

2.1 Identify Experiment Category

Choose the appropriate category: - cta - Call-to-action buttons and conversion elements - navigation - Navigation bars, menus, tabs - blog - Blog-specific layouts and features - visual - Colors, themes, visual styling - content - Content display and organization

2.2 Add to Config Template

Edit: projects/blog/docs/js/statsig/config/experiments.config.template.js

experiments: {
  cta: {
    // Existing experiments...

    // NEW EXPERIMENT
    hero_cta_style: {
      selector: "#hero-cta-btn",           // CSS selector for target element
      experiment: "blog_hero_cta_style",   // Statsig experiment name
      param: "button_style",               // Parameter name from Statsig
      type: "class",                       // Type of change (see types below)
      className: "md-button--primary",     // For type: "class"
      category: "conversion",              // Category for analytics
      pages: ["index.html"],               // Pages where this runs
      priority: "critical"                 // Priority level
    }
  }
}

2.3 Experiment Configuration Options

Required Fields

Field	Description	Example
`selector`	CSS selector for target element	`"#hero-cta"`
`experiment`	Statsig experiment name	`"blog_hero_cta_text"`
`param`	Parameter name from Statsig	`"button_label"`
`type`	Type of DOM manipulation	`"text"`
`category`	Analytics category	`"conversion"`
`pages`	Pages where experiment runs	`["index.html"]` or `["*"]`
`priority`	Importance level	`"critical"`

Experiment Types

1. Text Changes (type: "text")

{
  selector: "#hero-heading",
  experiment: "blog_hero_heading_text",
  param: "heading_text",
  type: "text",
  category: "conversion",
  pages: ["index.html"],
  priority: "high"
}

2. CSS Class Changes (type: "class")

{
  selector: "body",
  experiment: "blog_sticky_nav",
  param: "enabled",
  type: "class",
  className: "md-tabs--sticky",  // Class to add if param is truthy
  category: "navigation",
  pages: ["*"],
  priority: "medium"
}

3. Attribute Changes (type: "attr")

{
  selector: "[data-exp='pagination']",
  experiment: "blog_pagination_count",
  param: "count",
  type: "attr",
  attr: "data-pagination-count",  // Attribute to set
  category: "blog-layout",
  pages: ["blog/index.html"],
  priority: "low"
}

4. HTML Changes (type: "html")

{
  selector: ".blog-sidebar",
  experiment: "blog_sidebar_content",
  param: "html_content",
  type: "html",
  category: "content",
  pages: ["blog/*.html"],
  priority: "medium"
}

5. Style Changes (type: "style")

{
  selector: ".blog-card",
  experiment: "blog_card_layout",
  param: "layout_style",
  type: "style",
  styleProperty: "display",  // CSS property to change
  category: "visual",
  pages: ["blog/index.html"],
  priority: "low"
}

Optional Fields

Field	Description	Example
`fallback`	Default value if Statsig fails	`"Click Here"`
`className`	Class to add (for type: "class")	`"md-button--primary"`
`attr`	Attribute name (for type: "attr")	`"data-count"`
`styleProperty`	CSS property (for type: "style")	`"background-color"`

2.4 Update Production Config

In your GitHub Secrets, update EXPERIMENT_CONFIG_JSON_BLOG_PROD to include the new experiment.

Format (JSON):

{
  "experiments": {
    "cta": {
      "hero_cta_style": {
        "selector": "#hero-cta-btn",
        "experiment": "blog_hero_cta_style",
        "param": "button_style",
        "type": "class",
        "className": "md-button--primary",
        "category": "conversion",
        "pages": ["index.html"],
        "priority": "critical"
      }
    }
  }
}

Step 3: Prepare DOM Elements

3.1 Add Target Element

Make sure your HTML has an element matching the selector.

Example: Text Experiment

<!-- Add unique ID for targeting -->
<a href="/tutorial" id="hero-cta-btn" class="md-button" data-hidden>
  Read the Tutorial
</a>

3.2 Add Anti-Flicker Attribute

Critical: Add data-hidden to prevent content flash before variant loads.

<div id="experiment-target" data-hidden>
  Default content here
</div>

The CSS will hide this until the experiment loads:

[data-hidden] {
  visibility: hidden !important;
}

3.3 Add Tracking Attributes (Optional)

For custom click tracking:

<button id="hero-cta" data-track="custom_event_name" data-exp-key="hero_cta_text">
  Click Me
</button>

Attributes: - data-track - Custom event name for tracking - data-exp-key - Links click to experiment for experiment_interaction events

3.4 Page-Specific Experiments

Use the pages array to control where experiments run:

// Run only on homepage
pages: ["index.html"]

// Run on all blog posts
pages: ["blog/*.html"]

// Run everywhere
pages: ["*"]

// Run on specific pages
pages: ["index.html", "about.html", "blog/tutorial-1.html"]

Step 4: Test Locally

4.1 Enable Debug Logging

In your local experiments.config.js:

client: {
  options: {
    loggingLevel: "debug"  // Enable verbose logging
  }
}

4.2 Start Local Server

cd projects/blog
source ../../.venv/bin/activate
mkdocs serve

4.3 Test in Browser

Open: http://127.0.0.1:8000/
Open DevTools: Press F12 → Console tab

Expected Console Output:

[Statsig] Config loaded: {...}
[Statsig] Client initialized
[Statsig Renderer] Pre-hiding experiment elements
[Statsig Renderer] Applied experiment: hero_cta_text = "Start Learning Now"
[Statsig ExposureLogger] Logged exposure: blog_hero_cta_text = "Start Learning Now"
[Statsig EventLogger] Logged event: experiment_exposure

4.4 Test Both Variants

Option 1: URL Override

http://127.0.0.1:8000/?exp_blog_hero_cta_text=control
http://127.0.0.1:8000/?exp_blog_hero_cta_text=treatment

Option 2: Force in Statsig Console Go to Statsig Console → Experiments → Your Experiment → Overrides → Add your user ID

Option 3: Clear localStorage

// In browser console
localStorage.clear();
location.reload();

4.5 Test Tracking Events

Check exposure logging: - Should see experiment_exposure event when page loads

Check click tracking: - Click the experiment element - Should see experiment_interaction event with experiment metadata

Check Network tab: - Filter by "statsig" - Should see POST requests to api.statsig.com/v1/log_event

Click an internal link
Console should show: [Statsig Tracking] Page navigation detected, resetting trackers...
New exposure should be logged for the new page

4.7 Test Anti-Flicker

Disable cache: DevTools → Network → Disable cache
Slow 3G: DevTools → Network → Throttle to Slow 3G
Reload page: Should NOT see default content flash before variant

If you see a flash: - Check data-hidden attribute is on element - Check CSS includes anti-flicker rule - Check element selector is correct

Step 5: Deploy & Monitor

5.1 Commit Changes

git add projects/blog/docs/js/statsig/config/experiments.config.template.js
git add projects/blog/docs/index.md  # Or wherever you added HTML
git commit -m "feat: add hero CTA style experiment

- Add blog_hero_cta_style experiment to test button styling
- Target: #hero-cta-btn element
- Variants: primary vs secondary button style
- Expected lift: 15-20% CTR

Co-Authored-By: Claude Sonnet 4.5 <[email protected]>"

5.2 Update Production Config Secret

In GitHub: Settings → Secrets → Update EXPERIMENT_CONFIG_JSON_BLOG_PROD

Add your new experiment to the JSON.

5.3 Deploy to Staging (If Available)

git push origin develop
# Wait for GitHub Actions to deploy
# Test on staging URL

5.4 Deploy to Production

git checkout main
git merge feature_branch
git push origin main

5.5 Ramp Up in Statsig Console

Go to Statsig Console → Your Experiment
Start with 10% allocation
Monitor for 24 hours
If no issues, increase to 50%
Monitor until statistical significance

5.6 Monitor Metrics

Statsig Console: - Experiments → Your Experiment → Results - Check primary metric lift - Monitor secondary metrics - Watch for guardrail metrics (error rate, page load time)

Google Analytics (if enabled): - Events → Filter by statsig_experiment_exposure - Segments → Create segment by experiment variant - Compare metrics across variants

Key Metrics to Watch: - Click-through rate (primary for CTA tests) - Conversion rate - Time on page - Scroll depth - Bounce rate - Error rate (should not increase)

5.7 Statistical Significance

Wait until: - Minimum sample size: Usually 1000+ users per variant - Statistical significance: p-value < 0.05 (95% confidence) - Practical significance: Lift is meaningful (e.g., >10%) - Time window: At least 1-2 weeks to account for weekday/weekend differences

5.8 Make Decision

If Treatment Wins: 1. Increase allocation to 100% in Statsig 2. Update config to make treatment the default 3. Remove experiment code (now just standard code) 4. Archive experiment in Statsig

If Control Wins or No Difference: 1. Keep control (do nothing) 2. Archive experiment in Statsig 3. Remove experiment code 4. Document learnings

Common Experiment Patterns

Pattern 1: Button Text Optimization

Goal: Find the most compelling CTA text

Setup:

{
  selector: "#signup-cta",
  experiment: "blog_signup_cta_text",
  param: "button_text",
  type: "text",
  category: "conversion",
  pages: ["index.html"],
  priority: "critical"
}

Statsig Variants: - Control: "Sign Up" - Treatment_A: "Get Started Free" - Treatment_B: "Start Learning Now"

Success Metric: Click-through rate

Goal: Test sticky vs non-sticky navigation

Setup:

{
  selector: "body",
  experiment: "blog_sticky_nav",
  param: "enabled",
  type: "class",
  className: "md-tabs--sticky",
  category: "navigation",
  pages: ["*"],
  priority: "high"
}

Statsig Variants: - Control: false (no sticky nav) - Treatment: true (sticky nav)

Success Metrics: Page views per session, time on site

Pattern 3: Content Density

Goal: Find optimal number of posts per page

Setup:

{
  selector: "[data-exp='pagination']",
  experiment: "blog_posts_per_page",
  param: "count",
  type: "attr",
  attr: "data-posts-count",
  category: "blog-layout",
  pages: ["blog/index.html"],
  priority: "medium"
}

Statsig Variants: - Control: 5 - Treatment_A: 10 - Treatment_B: 3

Success Metrics: Engagement rate, pages per session

Pattern 4: Visual Styling

Goal: Test primary color variations

Setup:

{
  selector: ":root",
  experiment: "blog_primary_color",
  param: "color_hex",
  type: "style",
  styleProperty: "--md-primary-fg-color",
  category: "visual",
  pages: ["*"],
  priority: "low"
}

Statsig Variants: - Control: "#384D48" - Treatment_A: "#2196F3" - Treatment_B: "#4CAF50"

Success Metrics: Time on page, subjective feedback

Pattern 5: Multi-Element Test

Goal: Test entire page redesign

Approach: Use multiple coordinated experiments with the same allocation

experiments: {
  cta: {
    redesign_hero_cta: {
      selector: "#hero-cta",
      experiment: "blog_homepage_redesign_v2",
      param: "hero_cta_text",
      type: "text",
      // ...
    }
  },
  visual: {
    redesign_hero_style: {
      selector: ".hero-section",
      experiment: "blog_homepage_redesign_v2",
      param: "hero_style",
      type: "class",
      className: "hero--modern",
      // ...
    }
  }
}

Statsig: Use same experiment for both, different parameters

Experiment Best Practices

DO ✅

Start Small
Test one thing at a time
Small iterations are easier to understand
Define Success First
Know your primary metric before starting
Set a minimum detectable effect (MDE)
Test for Long Enough
Minimum 1-2 weeks
Wait for statistical significance
Monitor Guardrail Metrics
Page load time
Error rate
Core Web Vitals
Document Everything
Hypothesis
Expected lift
Results
Learnings
Use Anti-Flicker
Always add data-hidden to experiment elements
Test with slow network to verify
Test Both Variants
Manually verify both control and treatment work
Check on mobile and desktop

DON'T ❌

Don't Test Too Many Things
Running 10+ experiments simultaneously dilutes traffic
Harder to understand interactions
Don't Stop Too Early
Need statistical significance
Account for day-of-week effects
Don't Ignore Negative Results
Failed experiments teach you what doesn't work
Document and move on
Don't Break Existing Functionality
Always test both variants thoroughly
Monitor error rates
Don't Forget Mobile
Test on different screen sizes
Mobile users may behave differently
Don't Experiment on Critical Flows Without Testing
Thoroughly test payment, signup, login flows
Have rollback plan ready

Troubleshooting

Issue: Experiment Not Applying

Check: 1. Element selector is correct: document.querySelector("#your-selector") 2. Page matches pages array in config 3. Statsig experiment is running (not paused) 4. Browser console shows experiment loaded 5. Variant value is what you expect

Debug:

// In browser console
console.log(window.EXPERIMENT_CONFIG);
console.log(window.StatsigClient.instance);
console.log(window.StatsigClient.instance.getExperiment("your_experiment_name"));

Issue: Content Flash (FOUC)

Check: 1. Element has data-hidden attribute 2. Anti-flicker CSS is loaded 3. Scripts load in correct order (renderer before bootstrap)

Fix:

/* In extra.css */
[data-hidden] {
  visibility: hidden !important;
  opacity: 0 !important;
}

Issue: Events Not Logging

Check: 1. Tracking enabled in config: enableAutoClick: true 2. Element matches tracking selectors 3. Browser console shows tracking initialized 4. Network tab shows POST to Statsig API 5. Ad blocker not blocking requests

Debug:

// In browser console
console.log(window.StatsigTracker);
console.log(window.StatsigTracker.eventLogger);
console.log(window.StatsigTracker.clickTracker);

Issue: Wrong Variant Assigned

Check: 1. User ID is stable (check localStorage) 2. Allocation percentage in Statsig Console 3. No user overrides in Statsig Console

Reset:

// In browser console
localStorage.removeItem('statsig-user-id');
location.reload();

Issue: Experiment Not in Results

Check: 1. Exposures are being logged (check Statsig Events) 2. Enough time has passed (results update hourly) 3. Primary metric is configured correctly 4. Users are actually seeing both variants (check allocation)

Advanced Topics

Sequential Testing

Test multiple variations sequentially: 1. Test A vs B → B wins 2. Test B vs C → C wins 3. Test C vs D → C wins 4. C is the winner

Multi-Armed Bandit

Let Statsig automatically allocate more traffic to winning variant: - Enable "Auto-tune" in Statsig Console - System will gradually shift traffic to better performers

Personalization

Use Statsig targeting rules: - Show different variants based on: - Location - Device type - Time of day - User properties

Experiment Interactions

Be careful when running multiple experiments: - Experiments on same page may interact - Use Statsig's "Mutual Exclusion Groups" to prevent conflicts

Resources

Documentation

Statsig Docs: https://docs.statsig.com
This Implementation: PHASE3_IMPLEMENTATION.md
Technical Reference: tracking/README.md

Statsig Console

Experiments: https://console.statsig.com/experiments
Metrics: https://console.statsig.com/metrics
Events: https://console.statsig.com/events

Google Analytics (if enabled)

Events: GA4 → Reports → Engagement → Events
Explorations: GA4 → Explore → Create new exploration

Support

Statsig Community: https://statsig.com/slack
Documentation: Check implementation docs in this repo

Quick Reference

Experiment Config Template

experiment_name: {
  selector: "#element-id",              // Required: CSS selector
  experiment: "statsig_experiment_name", // Required: Statsig name
  param: "parameter_name",              // Required: Parameter from Statsig
  type: "text",                         // Required: text|class|attr|html|style
  category: "conversion",               // Required: Analytics category
  pages: ["index.html"],                // Required: Where to run
  priority: "critical",                 // Required: critical|high|medium|low
  fallback: "default value",            // Optional: Default if Statsig fails
  className: "css-class",               // For type: "class"
  attr: "attribute-name",               // For type: "attr"
  styleProperty: "css-property"         // For type: "style"
}

HTML Template

<element id="unique-id" data-hidden data-track="event_name" data-exp-key="config_key">
  Default content
</element>

Testing Checklist

[ ] Experiment created in Statsig Console
[ ] Config added to experiments.config.template.js
[ ] Production secret updated (EXPERIMENT_CONFIG_JSON)
[ ] HTML element has unique selector
[ ] Element has data-hidden attribute
[ ] Local testing passed (both variants work)
[ ] No content flash observed
[ ] Events logging correctly
[ ] Instant navigation works
[ ] Mobile testing passed
[ ] Deployed to staging
[ ] Monitored for 24h on staging
[ ] Deployed to production at 10% allocation
[ ] Ramped to 50% after validation
[ ] Monitoring results in Statsig Console

Last Updated: February 7, 2026 Version: 1.0.0

How to Add New A/B Tests - Step-by-Step Guide

Overview: The 5-Step Process

Step 1: Create Experiment in Statsig Console

1.1 Log into Statsig

1.2 Create New Experiment

1.3 Define Parameters

1.4 Set Success Metrics

1.5 Start Experiment

Step 2: Configure in Code

2.1 Identify Experiment Category

2.2 Add to Config Template

2.3 Experiment Configuration Options

Required Fields

Experiment Types

Optional Fields

2.4 Update Production Config

Step 3: Prepare DOM Elements

3.1 Add Target Element

3.2 Add Anti-Flicker Attribute

3.3 Add Tracking Attributes (Optional)

3.4 Page-Specific Experiments

Step 4: Test Locally

4.1 Enable Debug Logging

4.2 Start Local Server

4.3 Test in Browser

4.4 Test Both Variants

4.5 Test Tracking Events

4.6 Test Instant Navigation

4.7 Test Anti-Flicker

Step 5: Deploy & Monitor

5.1 Commit Changes

5.2 Update Production Config Secret

5.3 Deploy to Staging (If Available)

5.4 Deploy to Production

5.5 Ramp Up in Statsig Console

5.6 Monitor Metrics

5.7 Statistical Significance

5.8 Make Decision

Common Experiment Patterns

Pattern 1: Button Text Optimization

Pattern 2: Navigation Layout

Pattern 3: Content Density

Pattern 4: Visual Styling

Pattern 5: Multi-Element Test

Experiment Best Practices

DO ✅

DON'T ❌

Troubleshooting

Issue: Experiment Not Applying

Issue: Content Flash (FOUC)

Issue: Events Not Logging

Issue: Wrong Variant Assigned

Issue: Experiment Not in Results

Advanced Topics

Sequential Testing

Multi-Armed Bandit

Personalization

Experiment Interactions

Resources

Documentation

Statsig Console

Google Analytics (if enabled)

Support

Quick Reference

Experiment Config Template

HTML Template

Testing Checklist