Data engineering · plan by default

/data-backfill

Plan or run resumable historical-data backfills

Use for bounded historical reprocessing; data-pipeline creates normal transformation behavior, and a backfill that is a phase of a schema change belongs to db-migrate.

Make it your own.

In Claude Code, use the slash command and add your context. In Codex, select data-backfill from the just-vibe skill picker, then send the same brief.

Version 0.11.0 also supports /jv data-backfill, /just-vibe data-backfill and /jv:data-backfill in Claude. See shortcut setup and context examples.

Example · plan
/just-vibe:data-backfill Plan a resumable one-year backfill with explicit batch and load limits.
edge · plan
/just-vibe:data-backfill Plan resuming a backfill while the live pipeline updates the same historical records.
blocked · inspect
/just-vibe:data-backfill Plan a backfill without production scan permission or known source volume.

What the agent does

  1. Estimate volume and partition a fixed source snapshot.
  2. Define idempotent writes and checkpoints, and account for concurrent incremental writers.
  3. Measure and validate a small permitted batch, reconcile its output, then resume within limits.

Inputs

  • historical range, transformation/version, target, batch limits, and load budget.

Optional context: scope, references, constraints, successCriteria, environment, mode, budget.

Scope

Reads
Resumable historical recomputation; execution requires the specified data/environment authorization.
Writes
Inspect/plan: inspect or propose; save requested artifacts only. Apply: make the requested changes or execute the requested operation within its resolved target and limits. Local preparation does not authorize live, remote, destructive or paid actions; existing explicit session authorization still applies.
Mode
Plan the backfill; apply for requested backfill code or an explicitly scoped data run.
Prerequisites
Data source/version, schema/semantics, transformation code, permitted sampling scope, and storage/compute budget. Prefer aggregates and redacted samples; never upload datasets to external services implicitly. Record time zones and snapshot identity for reproducibility.

Expected output

  • Backfill plan or script, or a run record, with partitions, caps, checkpoints, reconciled counts, discrepancies and resume/recovery instructions.

How the work is checked

  • Restart skips or safely replays completed batches; live incremental processing does not race into inconsistent output.

When to stop or clarify

  • Pause on load/error thresholds. Never launch an unbounded production scan or overwrite historical data without explicit scope.

Handling missing context

Infer
Inspect schema, source snapshot, transformation code, grain, time zones and permitted sample scope.
Assume
Use bounded synthetic or supplied samples when full data is unavailable; keep unknown values distinct from zero.
Ask
Resolve ambiguous entity/grain/time semantics before reconciliation or backfill; obtain missing data/compute limits only for the dependent scan or execution.

Technical guidance

Evidence
Define bounded historical range, target revision, batch key, live-writer policy and resource limits.
Method
Make chunks resumable and idempotent; prevent old backfill data from overwriting newer live values.
Pitfall
A successful batch counter does not prove all rows were covered, and retrying non-idempotent transforms can corrupt values.
Check
Stop and resume midway while a live update occurs, then reconcile coverage and values at the agreed snapshot boundary.

Situational decisions

When a batch fails or load exceeds the cap: Pause with its partition identity and reconciliation state so resume cannot duplicate or overwrite good output.

When the request is for local preparation or implementation: Implement restartable batches, progress and reconciliation using a local fixture; resolve live target, limits and recovery before modifying existing data.

The coding agent follows this workflow using its available tools. Installation does not grant service access or guarantee an outcome. Read the compatibility notes.

Keep exploring