Spark
- Category
- Databases and data
- Docs outline
- 16 sections
Official docs outline
spark.apache.org- RDD Programming Guide
RDDs, transformations versus actions, lazy evaluation, shuffles, persistence, broadcast variables and accumulators.
- Spark SQL, DataFrames and Datasets
DataFrame API, Spark SQL, Catalyst optimizer and Datasets.
- Data Sources
Parquet, ORC, JSON, CSV, JDBC, partition discovery and save modes.
- Cluster Mode Overview
Driver, executors, cluster managers, jobs, stages and tasks.
- Submitting Applications
spark-submit, deploy modes, dependencies and master URLs.
- Performance Tuning
Caching, join strategy hints, broadcast joins, adaptive query execution, partition coalescing and skew joins.
- Tuning Spark
Memory management, serialization, garbage collection, parallelism and data locality.
- Configuration
Application properties, shuffle behavior, executor memory and dynamic allocation.
- Structured Streaming
Micro-batch processing, triggers, watermarks, windowing, output modes and checkpointing.
- Job Scheduling
Dynamic resource allocation, fair scheduler and resource sharing.
- Running on Kubernetes
Deploying Spark on Kubernetes, pods and resource requests.
- Monitoring and Instrumentation
Web UI, history server, metrics and event logs.
- MLlib
Pipelines, transformers, estimators and distributed machine learning.
- PySpark, Arrow and pandas
Python API, Arrow-optimized UDFs, pandas API on Spark.
- Security
Authentication, encryption and access control.
- Spark Connect
Client-server architecture for remote Spark access.
Each section links to the official page. Titles and summaries by Extra Credit; the docs belong to their authors.
