ChatGPT ab-test-plan skill for experiment metrics and decisions
Use baselines, targets, allocation, and statistical inputs to plan decision and guardrail metrics and how to interpret experiment outcomes.
中文Install in ChatGPT
A ChatGPT workspace with skill creation enabled. The example uses only the supplied plain-text material.
- In a ChatGPT workspace with skill creation enabled, open Plugins → Skills → Create → Create with editor.
- Create a skill in the editor and adapt the text-only `ab-test-plan` workflow from the pinned GitHub source into its instructions.
- Save the skill, then type @ in a new chat to select it and try the example.
Example task
Manually select ab-test-plan and use the fictional reading-list button copy inputs to design the experiment.
experiment `Reading list label` | control `Save` | variant `Save for later`
primary `completed saves` | guardrail `median page time`
baseline `20%` | target `24%` | allocation `50/50` | alpha `0.05` | power `0.80` | guardrail threshold `100 ms`
no daily eligible traffic estimate suppliedWhat to check
Use `completed saves` as the primary metric and compare `Save` with `Save for later`. Keep the `median page time` guardrail at an allowable increase of `100 ms`. Record the sample calculation method and inputs, and mark duration pending without daily eligible traffic.
Original SKILL.md on GitHub · Updated
Frequently asked question
Is a baseline and target enough to set the number of days?
Estimate samples per group from the design parameters, then use daily eligible traffic for duration. Mark duration pending until that traffic estimate is available.