Ciying AI Directory
AI DirectorySkills
中文
Ciying AI Directory

Paper Context Resolver: Check a Research Paper’s Dataset Split

Resolve one missing experimental detail by comparing repository documentation with the relevant original paper or supplement. Record the answer, evidence, and any mismatch before a replication run.

中文

Setup for Codex / ChatGPT (manual text adaptation)

Codex with the complete skill directory and the relevant source material. You can also try the text-only evidence comparison in a regular ChatGPT conversation using excerpts you provide.

  1. For Codex, copy the complete skills/paper-context-resolver directory from the pinned commit into .agents/skills/paper-context-resolver in your project. Keep the references directory and agents/openai.yaml.
  2. Start a Codex session from the project and choose paper-context-resolver in the skills picker. Provide the exact reproduction question and the repository and paper evidence.
  3. For the manual ChatGPT workflow, open a new conversation and paste the example with your own attributed excerpts. Include a section, table, or page reference whenever possible.

Example task

Resolve only which split produced the final accuracy in this fictional benchmark. Repository README: ‘Use dev.tsv in the quick-start command.’ Paper §4.2: ‘We report the final score on test.tsv; dev.tsv is for selecting hyperparameters.’ Repository results are silent about the final split. Give the answer, cite the supplied section, and annotate the distinction between the quick-start and the reported final evaluation. Do not run any commands.

Expected result

The answer cites paper section 4.2 for test.tsv and records that dev.tsv serves a separate purpose. It should mention that the supplied README command alone cannot establish how the final score was measured and attribute both facts to their respective excerpts.

Upstream documentation

Original SKILL.md on GitHub · MIT license · Source reviewed

Frequently asked question

Can Paper Context Resolver recreate an entire experiment on its own?

It answers one unresolved reproduction question, such as which split or preprocessing setting to use. The wider reproduction workflow can then use the evidence to decide what to run and note differences between the paper and repository.