Apache Spark is a multi-language engine for executing data engineering, data science, and machine learning on single-node machines or clusters.

If you’d like to build Spark from source, visit Building Spark. Spark runs on both Windows and UNIX-like systems (e.g. Linux, Mac OS), and it should run on any platform that runs a supported version of Java.

The documentation linked to above covers getting started with Spark, as well the built-in components MLlib, Spark Streaming, and GraphX. In addition, this page lists other resources for learning Spark.

Apache Spark started as a research project at UC Berkeley in the AMPLab, which focuses on big data analytics. Our goal was to design a programming model that supports a much wider class of …

Easy to use Spark Structured Streaming abstracts away complex streaming concepts such as incremental processing, checkpointing, and watermarks so that you can build streaming applications …

Jan 11, 2026 · We’re proud to announce the release of Spark 0.7.0, a new major version of Spark that adds several key features, including a Python API for Spark and an alpha of Spark Streaming.

Dec 11, 2025 · PySpark combines Python’s learnability and ease of use with the power of Apache Spark to enable processing and analysis of data at any size for everyone familiar with Python. PySpark …

We use Spark to regularly read raw data, convert them into Parquet, and process them to create advanced analytics dashboards: aggregation, sampling, statistics computations, anomaly detection, …

Spark SQL includes a cost-based optimizer, columnar storage and code generation to make queries fast. At the same time, it scales to thousands of nodes and multi hour queries using the Spark …

Spark SQL is the top active component in this release. 46% of the resolved tickets are for Spark SQL. These enhancements benefit all the higher-level libraries, including structured streaming and MLlib, …