Best for
- Optimizing slow Spark jobs
- Tuning memory and executor configuration
- Implementing efficient partitioning strategies
Source profileQuality 80/100
wshobson/agents/plugins/data-engineering/skills/spark-optimization/SKILL.md
spark-optimization
Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.
- Source repository stars
- 38,313
- Declared platforms
- 0
- Static risk flags
- 1
- Last source update
- 2026-07-22
- Source checked
- 2026-07-28
Decision brief
What it does—and where it fits
Production patterns for optimizing Apache Spark jobs including partitioning strategies, memory management, shuffle optimization, and performance tuning.
Not for
- Tasks that require unconfirmed production actions or broad system permissions.
- Environments where the pinned source and install steps cannot be inspected.
Compatibility matrix
Platform support, with evidence labels
| Platform | Status | Evidence | What to check |
|---|---|---|---|
| Codex | Not declared | No explicit evidence | Portability before use |
| Claude Code | Not declared | No explicit evidence | Portability before use |
| Cursor | Not declared | No explicit evidence | Portability before use |
| Gemini CLI | Not declared | No explicit evidence | Portability before use |
Installation
Inspect first. Install second.
The source command is displayed only when detected. A safe inspection prompt is always available so your agent can explain every action before execution.
Source-detected install commandSource
npx skills add https://github.com/wshobson/agents --skill "plugins/data-engineering/skills/spark-optimization"Safe inspection promptEditorial
Inspect the Agent Skill "spark-optimization" from https://github.com/wshobson/agents/blob/c4b82b0ad771190355eb8e204b1329732a18449a/plugins/data-engineering/skills/spark-optimization/SKILL.md at commit c4b82b0ad771190355eb8e204b1329732a18449a. List every install step, command, network request, credential, file read/write, external action, and rollback step. Explain whether it fits my task. Do not install or execute anything until I approve.
Workflow
What the source asks the agent to do
- 01
Quick Start
python from pyspark.sql import SparkSession from pyspark.sql import functions as F
python from pyspark.sql import SparkSession from pyspark.sql import functions as F - 02
When to Use This Skill
Optimizing slow Spark jobs
Optimizing slow Spark jobsTuning memory and executor configurationImplementing efficient partitioning strategies - 03
Core Concepts
Review the “Core Concepts” section in the pinned source before continuing.
Review and apply the “Core Concepts” source section. - 04
1. Spark Execution Model
Review the “1. Spark Execution Model” section in the pinned source before continuing.
Review and apply the “1. Spark Execution Model” source section.
Permission review
Static risk signals and limitations
Reads files
The documentation asks the agent to read local files, directories, or repositories.
Detailed pattern documentation lives in `references/details.md`. Read that file when the navigation tier above is insufficient.Evidence record
Why each signal appears
EvidenceSourceComputedTestedEditorial
| Signal | Value | Evidence type | Meaning |
|---|---|---|---|
| Quality score | 80/100 | Computed | Documentation, specificity, maintenance, and trust rules |
| Repository stars | 38,313 | Source | Repository attention, not individual Skill quality |
| Compatibility | 0 platforms | Source | Declared in the catalog source record |
| Usage guide | automated source guide | Editorial | Generated or reviewed according to the visible evidence level |
Pinned source
Provenance and original SKILL.md
- Repository
- wshobson/agents
- Skill path
- plugins/data-engineering/skills/spark-optimization/SKILL.md
- Commit
- c4b82b0ad771190355eb8e204b1329732a18449a
- License
- MIT
- Collected
- 2026-07-28
- Default branch
- main
View the original SKILL.md
Apache Spark Optimization
Production patterns for optimizing Apache Spark jobs including partitioning strategies, memory management, shuffle optimization, and performance tuning.
When to Use This Skill
- Optimizing slow Spark jobs
- Tuning memory and executor configuration
- Implementing efficient partitioning strategies
- Debugging Spark performance issues
- Scaling Spark pipelines for large datasets
- Reducing shuffle and data skew
Core Concepts
1. Spark Execution Model
Driver Program
↓
Job (triggered by action)
↓
Stages (separated by shuffles)
↓
Tasks (one per partition)
2. Key Performance Factors
| Factor | Impact | Solution |
|---|---|---|
| Shuffle | Network I/O, disk I/O | Minimize wide transformations |
| Data Skew | Uneven task duration | Salting, broadcast joins |
| Serialization | CPU overhead | Use Kryo, columnar formats |
| Memory | GC pressure, spills | Tune executor memory |
| Partitions | Parallelism | Right-size partitions |
Quick Start
from pyspark.sql import SparkSession
from pyspark.sql import functions as F
# Create optimized Spark session
spark = (SparkSession.builder
.appName("OptimizedJob")
.config("spark.sql.adaptive.enabled", "true")
.config("spark.sql.adaptive.coalescePartitions.enabled", "true")
.config("spark.sql.adaptive.skewJoin.enabled", "true")
.config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.config("spark.sql.shuffle.partitions", "200")
.getOrCreate())
# Read with optimized settings
df = (spark.read
.format("parquet")
.option("mergeSchema", "false")
.load("s3://bucket/data/"))
# Efficient transformations
result = (df
.filter(F.col("date") >= "2024-01-01")
.select("id", "amount", "category")
.groupBy("category")
.agg(F.sum("amount").alias("total")))
result.write.mode("overwrite").parquet("s3://bucket/output/")
Detailed patterns and worked examples
Detailed pattern documentation lives in references/details.md. Read that file when the navigation tier above is insufficient.
Best Practices
Do's
- Enable AQE - Adaptive query execution handles many issues
- Use Parquet/Delta - Columnar formats with compression
- Broadcast small tables - Avoid shuffle for small joins
- Monitor Spark UI - Check for skew, spills, GC
- Right-size partitions - 128MB - 256MB per partition
Don'ts
- Don't collect large data - Keep data distributed
- Don't use UDFs unnecessarily - Use built-in functions
- Don't over-cache - Memory is limited
- Don't ignore data skew - It dominates job time
- Don't use
.count()for existence - Use.take(1)or.isEmpty()