In the previous post shared how to use sbt in Spark-streaming project. This post is about how to create a fat jar for spark streaming project using sbt plugin. sbt-assembly is an sbt plugin to create a fat JAR of sbt project with all of its dependencies.
Add sbt-assembly plugin in project/plugin.sbt
Specify sbt-assembly.git as a dependency in project/project/build.scala
1 2 3 4 5 6 7
In build.sbt file add the following contents
1 2 3
Use full keys to configure the assembly plugin. For more details refer
1 2 3
If multiple files share the same relative path the default strategy is to verify that all candidates have the same contents and error out otherwise. This behaviour can be configured for Spark projects using assembly-merge-strategy as follows.
1 2 3 4 5 6 7 8 9
From the root folder run
the assembly plugin then packs the class files and all the dependencies into a single JAR file: target/scala_2.10/TwitterPopularTags-assembly-0.3.0.jar.
You can find an example project from here