PySpark SQL Cheatsheet
Quick reference guide for PySpark SQL, DataFrame APIs, and structured data query operations.
Initialize a SparkSession, the primary entry point for working with structured data (DataFrames and SQL) in Spark.
Create a DataFrame from an RDD by map-splitting lines and applying a Row schema structure (inferring schema).
Create a DataFrame from an RDD by programmatically specifying a Schema using StructType and StructField classes.
Read and load structured datasets from external files (JSON, Parquet, Text) into DataFrames.
Inspect basic attributes, view records, and check schema types of the DataFrame.
Retrieve summary statistics, count records, verify schema tree structure, and print execution query plans.
Project columns from the DataFrame. Supports basic column selecting and nested field expansions using explode().
Select columns and evaluate mathematical increments or boolean checks.
Evaluate condition checks inside queries, generating conditional outcomes using when-otherwise logic.
Keep rows matching a boolean criteria or match elements belonging to a specific list of values.
Query columns using text operations (regex-like, prefix/suffix searches, or substring slicing).
Filter or select values that fall within a lower and upper range boundaries (inclusive).
Add new columns or replace existing columns in the DataFrame with computed expressions.
Rename a column to a new target name, returning a new DataFrame projection.
Remove specified columns from the DataFrame by name or reference.
Deduplicate records in the DataFrame, returning only unique rows.
Group DataFrame rows by columns, allowing aggregations (like counting items per group).
Sort and order records by one or multiple columns, in ascending or descending sequence.
Handle missing datasets by filling defaults, dropping null rows, or swapping target values.
Register DataFrames as temporary tables, allowing SQL queries to be executed over them.
Execute SQL query strings programmatically, returning the query results as a new DataFrame.
Convert DataFrames into raw RDDs, JSON records arrays, or local Pandas DataFrames.
Manage partition counts across the cluster. repartition() performs full shuffles; coalesce() merges partitions without a shuffle.
Save DataFrame contents to target storage formats (Parquet, JSON, etc.) at path.
Close the SparkSession instance. Releases active context connections and cluster executor memory.