Codex Plan an A/B Test with Sample Size and Guardrail Rules
Specify a hypothesis, baseline, minimum detectable effect, randomization, and decision rules before an experiment.
中文Install in Codex
Use aggregated or synthetic metrics and provide baseline rate, eligible traffic, and maximum duration.
- Copy the pinned `skills/codexkit-a-b-test-planner/` directory into `.agents/skills/codexkit-a-b-test-planner/` in the project.
- Define the primary metric, sample-size assumptions, and guardrails in Codex, and ask for the plan only.
Example task
A fictional help-search completion rate is 32%, the target detectable lift is three percentage points, and eligible traffic is 500 users daily. Estimate days for a 50/50 user split and compare with a 14-day limit.What to check
The plan gives significance and power assumptions, sample size per arm, and whether 14 days supports the target detectable difference.
Original SKILL.md on GitHub · Updated
Frequently asked question
What should I do when 14 days is too short?
Report the power limits for that window and consider a longer run, a revised hypothesis, or more eligible traffic.