Performance tuning is an important aspect of working with Apache Spark, as it can help ensure that your data processing tasks are efficient and run smoothly. This comprehensive Apache Spark tutorial covers advanced performance optimization techniques that every data engineer should know. In this PySpark tutorial, we will delve into the five critical areas for Apache Spark performance tuning: spill, skew, shuffle, storage, and serialization. Whether you’re new to Apache Spark or looking to …

Two of the earlier posts are discussing installing Apache Spark-0.8.0 and Apache Spark-1.1.0 on Ubuntu-12.04 and Ubuntu-14.04 respectively. In this post you can discover necessary actions to set up Apache Spark-2.0.2 on Ubuntu 16.04 and Mac OS X Sierra. For enhanced guidance refer above mentioned posts.

Apache Spark is a powerful big data processing engine that is well-suited for use in a distributed environment. One way to interact with Spark is through the use of an IPython Notebook, which allows you to run and debug your Spark code in an interactive manner. This tutorial will guide you through the process of setting up and running a PySpark Notebook using Docker. Installing Docker Docker is a containerization platform that allows you to package and deploy your applications in a predictable …

In my previous post, I wrote about installation of Spark and Scala interactive shell. Here in this post, we’ll see how to do the same in Python. Similar to Scala interactive shell, there is an interactive shell available for Python. You can run it with the below command from spark root folder: ./bin/pyspark Now you can enjoy Spark using Python interactive shell. This shell might be sufficient for experimentations and developments. However, for production level, we should use a standalone …

Update: For Apache Spark 2 refer latest post One of the previous post mentioning about install Apache Spark-0.8.0 on Ubuntu-12.04. In this post explain about detailed steps to set up Apache Spark-1.1.0 on Ubuntu. For running Spark in Ubuntu machine should install Java. Using following commands easily install Java in Ubuntu machine. $ sudo apt-add-repository ppa:webupd8team/java $ sudo apt-get update $ sudo apt-get install oracle-java7-installer To check the Java installation is successful $ java …

In this post, we will discuss how to create an assembled JAR for a standalone Spark application using the sbt-assembly plugin. One of my previous posts, we discussed how to build a stand alone Spark Application using SBT eclipse plugin. Now, we will take it one step further and show you how to create a fat JAR for your Spark project using the sbt-assembly plugin. Adding the sbt-assembly Plugin The first step in creating an assembled JAR for your Spark application is to add the sbt-assembly …

This blog post will guide you through the process of building a Spark application in Scala that calculates popular hashtags from a Twitter stream. You will also learn how to use the sbt eclipse plugin to run the application in the Eclipse Integrated Development Environment (IDE). Whether you are new to big data processing or looking to improve your skills in data enginering and analytics, this tutorial has something to offer. Follow along with our step-by-step guide to develop your own stand …

Apache Spark has evolved dramatically since its early releases, becoming the de facto standard for large-scale data processing and analytics. This comprehensive guide covers installing the latest Apache Spark 3.5+ on modern Linux distributions with best practices for both development and production environments. Update Notice: This guide covers modern Apache Spark 3.5+ installation. For historical reference, our previous guides covered Apache Spark 1.0 installation and Apache Spark 2.x setup. …

You will need the following packages to run Mesos. $ sudo apt-get install python2.7-dev g++ libcppunit-dev libunwind7-dev git libcurl4-nss-dev You need to have Java installed, or the JAVA_HOME environment variable pointing to a Java installation. You can download the Mesos distribution from official website. After that untar the downloaded file $ tar xvf mesos-0.13.0.tar.gz Building and Installing $ cd mesos-0.13.0 $ mkdir build $ cd build $ sudo ../configure --prefix=/home/user/mesos $ sudo …

MQTT (Message Queuing Telemetry Transport) has become the backbone of IoT communication, enabling lightweight, reliable messaging between devices and applications. In this comprehensive tutorial, we’ll build a complete MQTT publisher-subscriber system in Scala using the Eclipse Paho library, perfect for IoT data streaming and distributed messaging architectures. Understanding MQTT: The IoT Messaging Standard MQTT is a publish-subscribe based lightweight messaging protocol designed …

Building web applications that handle dynamic, schema-less data often requires moving beyond traditional relational databases. In this tutorial, we’ll explore how to create a web-based paint application using Python Flask as our backend framework and MongoDB as our data storage solution. This post demonstrates the practical integration of Flask with MongoDB, showcasing why NoSQL databases are particularly well-suited for applications that handle varied, document-based data like our paint …

An application to draw simple drawings using lines, rectangles and circles in different colours. The application is developed using JavaScript and HTML5. The canvas feature in HTML5 is used for providing a drawable region. The JavaScript is used to handle drawing functions in this region. The select button to select the different tools to draw.

This blog post is an introduction to the world of GPU programming with CUDA. We will cover the fundamental concepts and tools necessary to get started with CUDA, including: The steps involved in a typical GPU program, such as allocating storage on the GPU, transferring data between the CPU and GPU, and launching kernels on the GPU to process the data. How to use the Nvidia C Compiler (nvcc) to compile CUDA code and follow conventions like naming GPU data with a “d” prefix. Key …

This post focuses on parallel computing on the GPU. Parallel computing is a way of solving large problems by breaking them into smaller pieces and run these smaller pieces at the same time. Main reasons of technical trends in the parallel computing on the GPU Modern processors are made from transistors. And each year, those transistors get smaller and smaller. The feature size is the minimum size of a transistor on a chip. As the feature size decreases, transistors get smaller, run faster, use …

HTML5 is the new HTML standard. One of the most interesting new features in HTML5 is the canvas element canvas for 2D drawing. A canvas is a rectangular area on an HTML page. All drawing on the canvas must be done using JavaScript. This post goes through the basics of implementing a 2D canvas context, and using the basic canvas functions for developing a simple game. Creating a canvad context, adding the canvas element to your HTML document like so 1<canvas id="Canvas" …