Creating Uber JARs for Spark Projects with sbt-assembly

sbt-assembly packages a Spark project plus all its dependencies into one runnable “uber” JAR, which you can hand to spark-submit without worrying about classpath. This follows on from the standalone Spark application in Scala post.

Adding the sbt-assembly plugin

The first step in creating an assembled JAR for your Spark application is to add the sbt-assembly plugin. To do this, you will need to add the following line to the project/plugin.sbt file:

1addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.9.1")

Configuring assembly settings

Next, you will need to specify sbt-assembly.git as a dependency in the project/project/build.scala file:

1import sbt._
2
3object Plugins extends Build {
4  lazy val root = Project("root", file(".")) dependsOn(
5    uri("git://github.com/sbt/sbt-assembly.git#0.9.1")
6  )
7}

In the build.sbt file, add the following contents:

1import AssemblyKeys._ // put this at the top of the file,leave the next line blank
2
3assemblySettings

You can use the full keys to configure the assembly plugin for more details refer

target                        assembly-jar-name             test
assembly-option               main-class                    full-classpath
dependency-classpath          assembly-excluded-files       assembly-excluded-jars

Configuring merge strategy

If multiple files share the same relative path, the default strategy is to verify that all candidates have the same contents and error out otherwise. This behavior can be configured for Spark projects using the assembly-merge-strategy as follows:

1mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
2  {
3    case PathList("javax", "servlet", xs @ _*) => MergeStrategy.last
4    case PathList("org", "apache", xs @ _*) => MergeStrategy.last
5    case PathList("com", "esotericsoftware", xs @ _*) => MergeStrategy.last
6    case "about.html" => MergeStrategy.rename
7    case x => old(x)
8  }
9}

Creating the fat JAR

Once you have added the sbt-assembly plugin and configured the assembly settings and merge strategy, you can create the fat JAR for your Spark application. From the root folder of your project, run the following command:

sbt/sbt assembly

This will create the JAR file in the target/scala_2.10/ directory. The name of the JAR file will be in the format of <ProjectName>-assembly-<version>.jar.

You can find an example project on how to create an assembled JAR for a Spark application on GitHub.

Once you have the fat JAR, see Apache Spark Performance Tuning for tuning the job it runs.