Best for
- Use when large data ingestion, backfill, export, ETL, warehouse loading, manifest catch-up, or table synchronization needs to become much faster while preserving data correctness.
Source profileQuality 76/100
affaan-m/ECC/skills/data-throughput-accelerator/SKILL.md
data-throughput-accelerator
Use when large data ingestion, backfill, export, ETL, warehouse loading, manifest catch-up, or table synchronization needs to become much faster while preserving data correctness.
- Source repository stars
- 234,327
- Declared platforms
- 0
- Static risk flags
- 0
- Last source update
- 2026-07-27
- Source checked
- 2026-07-28
Decision brief
What it does—and where it fits
Use this skill when the bottleneck is moving, transforming, or saving lots of data. The goal is not just speed. The goal is faster correct data landing in the right place with proof.
Not for
- Tasks that require unconfirmed production actions or broad system permissions.
- Environments where the pinned source and install steps cannot be inspected.
Compatibility matrix
Platform support, with evidence labels
| Platform | Status | Evidence | What to check |
|---|---|---|---|
| Codex | Not declared | No explicit evidence | Portability before use |
| Claude Code | Not declared | No explicit evidence | Portability before use |
| Cursor | Not declared | No explicit evidence | Portability before use |
| Gemini CLI | Not declared | No explicit evidence | Portability before use |
Installation
Inspect first. Install second.
The source command is displayed only when detected. A safe inspection prompt is always available so your agent can explain every action before execution.
Source-detected install commandSource
npx skills add https://github.com/affaan-m/ECC --skill "skills/data-throughput-accelerator"Safe inspection promptEditorial
Inspect the Agent Skill "data-throughput-accelerator" from https://github.com/affaan-m/ECC/blob/4e973d3eaf92d97f8d2e2d8abb39d8bdc8711b38/skills/data-throughput-accelerator/SKILL.md at commit 4e973d3eaf92d97f8d2e2d8abb39d8bdc8711b38. List every install step, command, network request, credential, file read/write, external action, and rollback step. Explain whether it fits my task. Do not install or execute anything until I approve.
Workflow
What the source asks the agent to do
- 01
Workflow
1. Read the current source, target, and manifest contracts. 2. Measure backlog: external files, manifest rows, raw rows, derived rows, min/max timestamps, and unprocessed counts. 3. Run a safe catch-up or sample benchmark. 4. Compare variants: batch size, worker count, warehouse…
Read the current source, target, and manifest contracts.Measure backlog: external files, manifest rows, raw rows, derived rows,Run a safe catch-up or sample benchmark. - 02
First Distinction
Separate these before optimizing:
source extraction speed;network transfer speed;warehouse/load speed; - 03
Fast Path Heuristics
Move compute to where the data already is.
Move compute to where the data already is.Prefer warehouse-native scans, joins, and appends for large landed files.Use manifests or checkpoints so completed files/partitions are skipped. - 04
Accounting Output
Use a hard accounting block:
Use a hard accounting block:
Permission review
Static risk signals and limitations
No configured static risk pattern was detected
This is not proof of safety. Runtime behavior, indirect dependencies, and hidden external systems are outside the static scan.
Evidence record
Why each signal appears
EvidenceSourceComputedTestedEditorial
| Signal | Value | Evidence type | Meaning |
|---|---|---|---|
| Quality score | 76/100 | Computed | Documentation, specificity, maintenance, and trust rules |
| Repository stars | 234,327 | Source | Repository attention, not individual Skill quality |
| Compatibility | 0 platforms | Source | Declared in the catalog source record |
| Usage guide | automated source guide | Editorial | Generated or reviewed according to the visible evidence level |
Pinned source
Provenance and original SKILL.md
- Repository
- affaan-m/ECC
- Skill path
- skills/data-throughput-accelerator/SKILL.md
- Commit
- 4e973d3eaf92d97f8d2e2d8abb39d8bdc8711b38
- License
- MIT
- Collected
- 2026-07-28
- Default branch
- main
View the original SKILL.md
Data Throughput Accelerator
Use this skill when the bottleneck is moving, transforming, or saving lots of data. The goal is not just speed. The goal is faster correct data landing in the right place with proof.
First Distinction
Separate these before optimizing:
- source extraction speed;
- network transfer speed;
- warehouse/load speed;
- transform speed;
- serving-table freshness;
- live tail growth while the job runs.
A pipeline can be "fast" and still appear behind if new data arrives faster than the final catch-up window.
Fast Path Heuristics
- Move compute to where the data already is.
- Prefer warehouse-native scans, joins, and appends for large landed files.
- Use manifests or checkpoints so completed files/partitions are skipped.
- Use partitioning and clustering that match the read and append pattern.
- Batch small files, requests, and writes.
- Make writes idempotent through unique keys, manifests, or replaceable staging.
- Keep raw, derived, and serving tables separately accountable.
Workflow
- Read the current source, target, and manifest contracts.
- Measure backlog: external files, manifest rows, raw rows, derived rows, min/max timestamps, and unprocessed counts.
- Run a safe catch-up or sample benchmark.
- Compare variants: batch size, worker count, warehouse SQL, file grouping, staging shape, and manifest update method.
- Promote only the fastest path that keeps counts and timestamps coherent.
- Codify the path as a CLI, scheduled job, workflow, or runbook.
- Rerun final accounting after the codified path executes.
Accounting Output
Use a hard accounting block:
Data throughput result:
- Source files discovered: 294
- Files processed this run: 294
- Raw rows added: 9,683,598
- Derived rows added: 8,917,585
- Remaining tail: 24 files at readback time
- Runtime: 38.7s
- Correctness gate: manifest counts and table max timestamps match
Guardrails
- Do not delete raw data to make a metric look better.
- Do not skip failed files silently.
- Do not mix historical backfill status with live-tail freshness.
- Do not call a pipeline complete until the target tables and manifest agree.
- For finance, healthcare, regulated, or customer-impacting data, preserve replay evidence and approval gates.
Alternatives
Compare before choosing
Computed 997
event4u-app/agent-config
design-review
Use when the user says "review the design", "check the UI", or wants a comprehensive UI/UX review. Uses a 7-phase methodology covering interaction, responsiveness, accessibility, and more.
Computed 9831,966
K-Dense-AI/scientific-agent-skills
dask
Distributed computing for larger-than-RAM pandas/NumPy workflows. Use when you need to scale existing pandas/NumPy code beyond memory or across clusters. Best for parallel file processing, distributed ML, integration with existing pandas code. For out-of-core analytics on single machine use vaex; for in-memory speed use polars.
Computed 9831,966
K-Dense-AI/scientific-agent-skills
medchem
Medicinal chemistry filters for compound triage. Apply drug-likeness rules (Lipinski, Veber, CNS), structural alert catalogs (PAINS, NIBR, ChEMBL), complexity metrics, and the medchem query language for library filtering.
Computed 9831,966
K-Dense-AI/scientific-agent-skills
neurokit2
Use NeuroKit2 to build or audit reproducible research workflows for physiological time-series preprocessing, event/interval analysis, multimodal alignment, variability, and complexity. Trigger when code imports neurokit2 or needs its current APIs, schemas, and method-aware validation—not for diagnosis or device validation.