A/B Test Analysis Skill for Codex

Analyze control and treatment samples with conversion rates, absolute differences, relative lift, and confidence intervals. Check traffic allocation and guardrails before interpreting the outcome.

中文

Install in Codex

Use a Codex environment that can discover project skills. The example can be tested in a local project. A Python environment is needed if Codex runs a statistical script.

In a temporary directory, run git clone https://github.com/bssm-oss/pm-skills-codex.git. Enter the clone, run git checkout --detach 9f19e3ff563a1cb4024cfb9a290d6002607b9dc9, and review the pinned SKILL.md.
Copy the entire pm-data-analytics/skills/ab-test-analysis folder to .agents/skills/ab-test-analysis in your own project, keeping its scripts, templates, and references together.
Open Codex in that project. Choose the skill with /skills or mention $ab-test-analysis, then run the example below. Restart Codex if the list does not refresh.

Example task

Use two groups of 2,000 users with 140 control conversions and 172 variant conversions. Request a two-sided proportions test, absolute and relative lift, a 95% confidence interval, and a note that no guardrails were supplied.

What to check

The conversion rates are 7.0% and 8.6%, a 1.6 percentage-point difference and about 22.9% relative lift. Report uncertainty if the interval crosses zero, and record the missing guardrails.

Original SKILL.md on GitHub · Updated 2026-11-11

Frequently asked question

Is a higher conversion rate enough to call a test successful?

Review sample size, allocation, uncertainty, and guardrails first. Use the estimated effect and its business implications to decide whether more evidence is needed.