This is One of the Solutions

Creating Assembled JAR for Standalone Spark Application

| Comments

In the previous post shared how to use sbt in Spark-streaming project. This post is about how to create a fat jar for spark streaming project using sbt plugin. sbt-assembly is an sbt plugin to create a fat JAR of sbt project with all of its dependencies.

Add sbt-assembly plugin in project/plugin.sbt

1
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.9.1")

Specify sbt-assembly.git as a dependency in project/project/build.scala

1
2
3
4
5
6
7
import sbt._

object Plugins extends Build {
  lazy val root = Project("root", file(".")) dependsOn(
    uri("git://github.com/sbt/sbt-assembly.git#0.9.1")
  )
}

In build.sbt file add the following contents

1
2
3
import AssemblyKeys._ // put this at the top of the file,leave the next line blank

assemblySettings

Use full keys to configure the assembly plugin. For more details refer

1
2
3
target                        assembly-jar-name             test
assembly-option               main-class                    full-classpath
dependency-classpath          assembly-excluded-files       assembly-excluded-jars

If multiple files share the same relative path the default strategy is to verify that all candidates have the same contents and error out otherwise. This behaviour can be configured for Spark projects using assembly-merge-strategy as follows.

1
2
3
4
5
6
7
8
9
mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
  {
    case PathList("javax", "servlet", xs @ _*) => MergeStrategy.last
    case PathList("org", "apache", xs @ _*) => MergeStrategy.last
    case PathList("com", "esotericsoftware", xs @ _*) => MergeStrategy.last
    case "about.html" => MergeStrategy.rename
    case x => old(x)
  }
}

From the root folder run

1
sbt/sbt assembly

the assembly plugin then packs the class files and all the dependencies into a single JAR file: target/scala_2.10/TwitterPopularTags-assembly-0.3.0.jar.

You can find an example project from here

Related posts

Comments