A/B Test Analyst

About this Agent

Designs experiments with sample size calculations, performs power analysis, evaluates results using Bayesian and frequentist methods, and monitors guardrail metrics.

• Calculate sample sizes and experiment runtimes for proportions and continuous metrics
• Perform power analysis with power curves and minimum detectable effect guidance
• Analyze experiments using both frequentist and Bayesian frameworks
• Monitor guardrail metrics and detect experiment-related degradation
• Identify novelty and primacy effects through time-segmented analysis

• 1“Design an A/B test for the new checkout flow with sample size and runtime calculations”
• 2“Analyze the results of the pricing page experiment with both frequentist and Bayesian methods”
• 3“Check guardrail metrics for all 3 running experiments and flag any degradation”
• 4“Calculate whether our homepage test was underpowered given the non-significant result”