A walkthrough of installing Apache Spark 3.5 on modern Linux, from prerequisites through a working standalone cluster. Earlier versions are covered in Install Apache Spark 1.0 on Ubuntu 14.04 and Install Apache Spark 2 on Ubuntu 16.04 and macOS. Prerequisites Requirement Version / Recommendation Java OpenJDK 17 (Spark 3.5 supports 8, 11, 17) Python 3.8+ for PySpark Memory 4 GB minimum, 8 GB+ for comfortable work Storage 10 GB free for install + logs OS Ubuntu 20.04+, CentOS 7+, or equivalent …

Performance tuning decides whether a Spark job runs in 10 minutes or 10 hours. Most slowdowns you’ll hit in production come from the same five areas: spill, skew, shuffle, storage, and serialization. This guide walks through each one with the cause, how to spot it in the Spark UI, and the PySpark code to fix it. The examples use PySpark, but the concepts apply to Scala and Java Spark equally well.

Earlier posts covered Spark 1.1.0 on Ubuntu 14.04. This one walks through Spark 2.0.2 on Ubuntu 16.04 and Mac OS X Sierra. For the latest version, see Install Apache Spark 3.5 on Linux.

Apache Spark works well in a Jupyter notebook: you get iterative development, inline plots, and the ability to poke at intermediate DataFrames. Docker makes the setup reproducible and removes the “works on my machine” problem. This post walks through running PySpark in Jupyter via the official jupyter/pyspark-notebook image. Installing Docker Docker is a containerization platform that allows you to package and deploy your applications in a predictable and isolated environment.

The earlier install post covered the Scala interactive shell. This one shows how to do the same in Python, then how to turn an experiment into a standalone application you can run with spark-submit. ./bin/pyspark Now you can enjoy Spark using Python interactive shell. This shell might be sufficient for experimentations and developments. However, for production level, we should use a standalone application.

Update: For Spark 2 see the Ubuntu 16.04 and macOS post. This post walks through Spark 1.1.0 on Ubuntu 14.04. For the latest version, see Install Apache Spark 3.5 on Linux. Prerequisites Start with Java: $ sudo apt-add-repository ppa:webupd8team/java $ sudo apt-get update $ sudo apt-get install oracle-java7-installer To check that the Java installation is successful: $ java -version It shows installed java version java version "1.7.0_72"_ Java(TM) SE Runtime Environment (build …

sbt-assembly packages a Spark project plus all its dependencies into one runnable “uber” JAR, which you can hand to spark-submit without worrying about classpath. This follows on from the standalone Spark application in Scala post. Adding the sbt-assembly plugin The first step in creating an assembled JAR for your Spark application is to add the sbt-assembly plugin. To do this, you will need to add the following line to the project/plugin.sbt file: …

This post walks through building a Spark Streaming application in Scala that extracts popular hashtags from the Twitter firehose, packaged with sbt, and runnable from the Eclipse IDE via the sbteclipse plugin. Building a Spark application using SBT A standalone Scala application built against the Apache Spark API and packaged with sbt (Simple Build Tool). For creating a stand alone app take the twitter popular tag example This program calculates popular hashtags (popular topics) over sliding 10 …

Apache Mesos is a cluster manager that abstracts CPU, memory, and storage across machines so frameworks (like Spark, Marathon, or Chronos) can run workloads without worrying about which physical node they land on. This post covers installing 0.13 on Ubuntu 12.04 and bringing up a minimal master + slave cluster. Prerequisites Install the build dependencies and make sure Java is available: $ sudo apt-get install python2.7-dev g++ libcppunit-dev libunwind7-dev git libcurl4-nss-dev You need to have …

MQTT is a lightweight publish-subscribe protocol widely used in IoT and telemetry because it runs fine over slow, lossy networks. This post builds a minimal publisher and subscriber in Scala using the Eclipse Paho library, talking to a local Mosquitto broker. The complete code is on GitHub. How MQTT works MQTT uses a central broker. Publishers send messages to topics, and any subscribers listening to a matching topic pattern receive them. Topics are hierarchical strings separated by /, for …

Schema-less, document-shaped data like freehand drawings is a good fit for MongoDB: there’s no rigid table to design around stroke coordinates that vary per drawing. This post ties the canvas front-end to a Flask backend that persists drawings in MongoDB. How the app works The user draws lines, rectangles, or circles on an HTML5 canvas. Each shape is stored in a JavaScript object with its coordinates and colour. When the user clicks “save”, the frontend POSTs the entire drawing …

A small browser-based drawing tool built on the HTML5 <canvas> element and plain JavaScript. How it works The page contains a <canvas> element that listens for three mouse events: mousedown, mousemove, and mouseup. When you press and drag, the handler draws the currently selected shape (line, rectangle, or circle) in the chosen colour. A toolbar above the canvas lets you pick the tool and colour. All drawing state lives in the browser; nothing hits the server until you explicitly …

Following on from my introduction to parallel programming, this post walks through a simple CUDA program that computes the squares of 64 numbers on the GPU. The source is on GitHub. A typical GPU program A typical CUDA program follows four steps: CPU allocates storage on the GPU CPU copies input data from CPU to GPU CPU launches kernels on the GPU to process the data CPU copies the result back from GPU to CPU Compiling 1nvcc -o square square.cu Instead of the regular C compiler, we use nvcc, the …

This post focuses on parallel computing on the GPU. Parallel computing solves large problems by breaking them into smaller pieces and running those pieces at the same time. Why GPUs for parallel computing Modern processors are made from transistors. Each year those transistors get smaller. The feature size is the minimum size of a transistor on a chip. As it decreases, transistors get smaller, run faster, use less power, and you can fit more of them on a chip, giving more resources for …

HTML5 introduced the <canvas> element for 2D drawing: a rectangular area on the page that JavaScript can paint into. This post walks through building a simple bird-shooting game using canvas drawing, setInterval timing, keyboard events, and basic collision detection. Add a canvas element to the HTML document: 1<canvas id="Canvas" width="800" height="450"></canvas> To draw inside the canvas, use JavaScript. First find the canvas element using …