Claude Code daft-distributed-scaling skill for partitioning and batching
Choose repartitioning or batching based on data volume and available workers. Observe memory use and scheduling overhead.
中文Install in Claude Code
A local Claude Code project with Daft configured. Use an available Ray environment to verify repartitioning across nodes. Inspect batching with a small fixture first.
- Review `.claude/skills/daft-distributed-scaling/SKILL.md` and its requirements at the pinned GitHub commit.
- Create `.claude/skills/daft-distributed-scaling/` in your project and save the reviewed file as `.claude/skills/daft-distributed-scaling/SKILL.md`.
- Start Claude Code from that project's root, then select `/daft-distributed-scaling` in the slash menu.
Example task
Invoke /daft-distributed-scaling and propose a stage-by-stage trial for this fictional thumbnail workload.
workload `album thumbnail export` | entries metadata first then image decode
light stage `path metadata` | heavy stage `image decode`
constraints `4 workers`, small pilot with each chunk `32 rows`What to check
Propose testing repartitioning under Ray for `path metadata`, then `into_batches(32)` for `image decode`. Check output with chunks of `32 rows` and record memory and time.
Original SKILL.md on GitHub · Updated
Frequently asked question
When should I favor smaller batches?
Use batching first for rows that take substantial memory. Adjust the batch size using observed memory and runtime to account for scheduling overhead.